A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature

By Benjamin E. Nye, Junyi Jessy Li, Roma Patel, Yinfei Yang, Iain James Marshall, Ani Nenkova, Byron C. Wallace. 2018

254 citations recorded by the source corpus.

We present a corpus of 5,000 richly annotated abstracts of medical articles describing clinical randomized controlled trials. Annotations include demarcations of text spans that describe the Patient population enrolled, the Interventions studied and to what they were Compared, and the Outcomes measured (the ‘PICO’ elements). These spans are further annotated at a more granular level, e.g., individual interventions within them are marked and mapped onto a structured medical vocabulary. We acquired annotations from a diverse set of workers with varying levels of expertise and cost. We describe our data collection process and the corpus itself in detail. We then outline a set of challenging NLP tasks that would aid searching of the medical literature and the practice of evidence-based medicine.

Subfields: Artificial Intelligence, Molecular Biology, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Natural Language Processing

Citations in the corpus

SciBERT: A Pretrained Language Model for Scientific Text (2019, background): “In addition, S CI B-ERT achieves new SOTA results on BC5CDR and ChemProt (Lee et al., 2019), and EBM-NLP (Nye et al., 2018).”
Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing (2020, methodology): “The focus has evolved from simple tasks, such as named entity recognition, to more sophisticated tasks, such as relation extraction and question answering, and new tasks have been proposed for emerging application scenarios such as evidence-based medical information extraction [43].”
SciBERT: Pretrained Contextualized Embeddings for Scientific Text (2019, unknown): “In addition, SCIBERT achieves new state-of-the-art (SOTA) results on the following datasets: BC5CDR (Yoon et al., 2018), EBM-NLP (Nye et al., 2018), and ChemProt (Lim and Kang, 2018).”
Fact or Fiction: Verifying Scientific Claims (2020, methodology): “, 2017) seeks to automate the process of creating systematic reviews of the medical literature7 – for instance, by extracting PICO snippets (Nye et al., 2018) and inferring the outcomes of clinical trials (Lehman et al.”
MSˆ2: Multi-Document Summarization of Medical Studies (2021, methodology): “Using the EBM-NLP corpus (Nye et al., 2018), a crowd-sourced collection of PIO tags,4 we train a token classification model (Wolf et al.”
A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, background): “The results illustrated that InstructGPT can perform very well on biomedical evidence extraction [74], medication status extraction [75], and medication attribute extraction [75].”
Large language models are few-shot clinical information extractors (2022, methodology): “, 2016) and to identify of pairs of spans with redundant information (Nye et al., 2018).”
OAG-BERT: Pre-train Heterogeneous Entity-augmented Academic Language Models (2021, background): “Some datasets like BC5CDR [26] only need span range identification while other datasets like EBM-NLP [33] also need entity type recognition.”
BERT: A Review of Applications in Natural Language Processing and Understanding (2021, methodology): “Datasets from subject areas were used: biomedicine (ChemProt [31], EBM-NLP [57] and others), computer science (SciERC [47], ACL-ARC [28]), and interdisciplinary (SciCite, described by A.”
Toward systematic review automation: a practical guide to using machine learning tools in research synthesis (2019, background): “Recently, Nye et al. released the EBM-NLP dataset [23], which comprises ~ 5000 abstracts of RCT reports manually annotated in detail.”