The NLM Colloquia on Biomedical Data Science and Computational Biology Research is a regularly scheduled series of scientific lectures presented by the NLM Division of Intramural Research (DIR), a premier hub of innovation for computational biology and biomedical data science.
The NLM DIR invites experts from outside NLM to present at the Colloquia, where they can share their insights with research communities across NIH and worldwide in the rapidly evolving fields of biomedical data science and computational biology research, as well as how their work impacts these topic areas.
2025 NLM Ada Lovelace Computational Health Lecture
The National Library of Medicine (NLM) is pleased to announce the 2025 NLM Ada Lovelace Computational Health Lecture. This lecture series, presented by NLM’s Division of Intramural Research, recognizes the contributions of computer scientists in research on health and biomedicine. The series features women in STEM sectors and invites them to share their pioneering research with NIH and beyond.
Evaluating Messy Tasks
Event Date: Monday, December 15, 2025
Time: 11:00am–12:00pm
Speaker: Ellen Voorhees , PhD
Location: NIH Natcher Building (45), Rm E1/E2, and virtual via NIH Videocast
Abstract:
A messy task is one in which humans have valid differences of opinion as to the quality or correctness of a candidate response. Language processing is rife with messy tasks---humans disagree about the relevance of documents retrieved in response to query, the acceptability of answers to a question (even for factoid questions), the quality of a document summary, the appropriateness of a translation, and the helpfulness of an explanation to name a few. Evaluating automated systems for messy tasks is complicated by this lack of a true gold standard.
The Text REtrieval Conference (TREC, trec.nist.gov) is a community evaluation that has been defining evaluations for information access tasks for more than thirty years. This talk will use TREC tracks to illustrate how humans' inherent disagreement has shaped the evaluation methodologies that drive technology development for messy language tasks, and to explore the implications for evaluating (even messier) AI tasks.
Speaker Bio:
Ellen Voorhees, PhD is a scientist emeritus in the Information Technology Laboratory at the U.S. National Institute of Standards and Technology (NIST). Her primary responsibility at NIST was managing the Text REtrieval Conference (TREC), an international workshop series that defines the methodology and builds the infrastructure required for large-scale evaluation of search and related technologies. Her research focuses on developing and validating appropriate evaluation schemes to measure system effectiveness in these areas. Voorhees has been elected into the NIST Portrait Gallery, and is a fellow of the ACM, an inaugural member of the ACM SIGIR Academy, and the 2024 recipient of the ACM SIGIR Gerard Salton award.
How to Join:
Location: NIH Natcher Building (45), Rm E1/E2
This talk will also be broadcast live: NIH Videocast
Interpreting services are available upon request. Individuals with disabilities who need reasonable accommodation to participate in this lecture should contact NLMColloquia@nih.gov or the Federal Relay (1-800-877-8339).
Questions during the presentation can be sent to: NLMColloquia@nih.gov.
Sponsored by:
Richard Scheuermann, PhD
Scientific Director, Division of Intramural Research, National Library of Medicine