ICAIL 2025 Tutorial: Technology-Assisted Review for High Recall Retrieval

Technology-assisted review (TAR) refers to iterative workflows that combine human review with AI techniques such as active learning and LLMs to minimize both time and manual effort while maximizing effectiveness. The use of TAR in the discovery process in civil litigation is a multi-billion dollar industry which has had an enormous impact on the practice of law. This application of TAR, along with applications to internal investigations and sunshine law requests, constitute the largest and most well-established application of AI in the law. The history of TAR rollout also has lessons for the adoption of AI technology more broadly in the law. The morning portion of the tutorial will cover key concepts in TAR, an overview of the technologies and workflow designs used, the basics of practical evaluation methods, and legal and ethical implications of TAR deployment. The afternoon will go into more technical depth on the implications of TAR workflows for supervised learning algorithm design, how generative AI is being applied in TAR, more sophisticated evaluation techniques (including for generative AI), and a wide range of open research questions.

Details TBA

Presenters

Lenora Gray is a Data Scientist at Elevate, with over 15 years of expertise in legal project management, eDiscovery, and data analytics. She has degrees in computer science and data science, and extensive experience in the execution and evaluation of TAR workflows. She received a best paper award at the 2023 International Conference on Artificial Intelligence and Law for a paper on TAR.

**David D. Lewis is Chief Scientific Officer at Nextpoint. He has researched and implemented TAR systems since the early 1990’s. He is a frequent public speaker, has taught over 25 tutorials to widely varying audiences, developed and taught the first graduate computer science class on eDiscovery, and has testified in US Federal court about TAR systems. He is a Fellow of the American Association for the Advancement of Science.

Jeremy Pickens is a pioneer in the field of collaborative exploratory search, a form of information seeking in which a group of people who share a common information need actively collaborate to achieve it. His ongoing research and development at Elevate focuses on methods for continuous learning, explicit user feedback and interaction design, and process-oriented approaches to complex, long-running information needs. Dr. Pickens earned his doctoral degree at the University of Massachusetts, Amherst, Center for Intelligent Information Retrieval. Before joining Catalyst Repository Systems and later OpenText, he spent five years as a research scientist at FX Palo Alto Lab, Inc.

Eugene Yang is a Research Scientist at the Human Language Technology Center of Excellence at Johns Hopkins University. Eugene received his Ph.D. from Georgetown University, where he worked on cost analysis and stopping rules for High Recall Retrieval. His work has influenced the design of retrieval systems used by law firms and legal service providers in the United States. Besides HRR, Eugene also works on cross-language and multilingual ad-hoc retrieval. He received the best student paper award at the 2021 ACM Document Engineering Conference for a paper on TAR. He offered the TAR Tutorial at ECIR 2022 and Neural Cross-Language IR at SIGIR 2023. He also frequently delivers guest lectures on CLIR and neural IR