Assessing the state of the art in biomedical relation extraction: overview of the BioCreative V chemical-disease relation (CDR) task

By Chih-Hsuan Wei, Yifan Peng, Robert Leaman, Allan Peter Davis, Carolyn J. Mattingly, Jiao Li, Thomas C. Wiegers, Zhiyong Lu. 2016

357 citations recorded by the source corpus.

Manually curating chemicals, diseases and their relationships is significantly important to biomedical research, but it is plagued by its high cost and the rapid growth of the biomedical literature. In recent years, there has been a growing interest in developing computational approaches for automatic chemical-disease relation (CDR) extraction. Despite these attempts, the lack of a comprehensive benchmarking dataset has limited the comparison of different techniques in order to assess and advance the current state-of-the-art. To this end, we organized a challenge task through BioCreative V to automatically extract CDRs from the literature. We designed two challenge tasks: disease named entity recognition (DNER) and chemical-induced disease (CID) relation extraction. To assist system development and assessment, we created a large annotated text corpus that consisted of human annotations of chemicals, diseases and their interactions from 1500 PubMed articles. 34 teams worldwide participated in the CDR task: 16 (DNER) and 18 (CID). The best systems achieved an F-score of 86.46% for the DNER task—a result that approaches the human inter-annotator agreement (0.8875)—and an F-score of 57.03% for the CID task, the highest results ever reported for such tasks. When combining team results via machine learning, the ensemble system was able to further improve over the best team results by achieving 88.89% and 62.80% in F-score for the DNER and CID task, respectively. Additionally, another novel aspect of our evaluation is to test each participating system’s ability to return real-time results: the average response time for each team’s DNER and CID web service systems were 5.6 and 9.3 s, respectively. Most teams used hybrid systems for their submissions based on machining learning. Given the level of participation and results, we found our task to be successful in engaging the text-mining research community, producing a large annotated corpus and improving the results of automatic disease recognition and CDR extraction. Database URL: http://www.biocreative.org/tasks/biocreative-v/track-3-cdr/

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Molecular Biology, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Molecular Biology

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Snorkel: Rapid Training Data Creation with Weak Supervision (2017, methodology): “Chemical-Disease Relations (CDR): We used the 2015 BioCreative chemical-disease relation dataset [49], where the task is to identify mentions of causal links between chemicals and diseases in PubMed abstracts.”
Training Classifiers with Natural Language Explanations (2018, background): “Sentences and ground truth labels came from a portion of the 2015 BioCreative chemical-disease relation dataset (Wei et al., 2015), which contains abstracts from PubMed.”
Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Health (2023, methodology): “…prompt engineering method to evaluate ChatGPT’s performance on biomedical NER and RE in the BLURB benchmark datasets including BC5CDR-chemical [94], BC5CDR-disease [94], NCBI-disease [99], BC2GM [100], JNLPBA [101], ChemProt [102], DDI [103], and GAD [104] in a zero-shot or few-shot manner.”
On the Accuracy of Influence Functions for Measuring Group Effects (2019, methodology): “The sentences and ground truth labels were sourced from the 2015 BioCreative chemical-disease relation dataset (Wei et al., 2015).”
Socratic Learning (2016, background): “[12], where mentions of diseases are extracted from PubMed abstracts and the labeling functions are written by a team of domain experts.”
BioCreative V CDR task corpus: a resource for chemical disease relation extraction (2016, methodology): “CTD triaged and/or curated 143 articles in conjunction with BioCreative V; the final 100 selected for inclusion in the Test Dataset represented abstract-only curation for CDRs.”
Slice-based Learning: A Programming Model for Residual Learning in Critical Data Slices (2019, methodology): “We consider two text-based relation extraction datasets: Chemical-Disease Relations (CDR),[41], in which we identify causal links between chemical and disease entities in a dataset of PubMed abstracts, and Spouses [9], in which we identify mentions of spousal relationships using preprocessed pairs o”
Socratic Learning: Augmenting Generative Models to Incorporate Latent Subsets in Training Data (2016, methodology): “We applied Socratic learning to two text-based relation extraction tasks, Disease Tagging (DT) and Chemical Disease Relation Extraction (CDR) [30], which look for mentions of diseases and valid disease-chemical pairs in PubMed abstracts.”
Snuba: Automating Weak Supervision to Label Training Data (2018, methodology): “For CDR, users relied on distant supervision through the Comparative Toxicogenomics Database [10].”
Deep learning for extracting protein-protein interactions from biomedical literature (2017, background): “…of biomedical information available in the textual form, there has been considerable interest in applying natural language processing (NLP) techniques and machine learning (ML) methods to the biomedical literature (Huang and Lu, 2015; Leaman and Lu, 2016; Singhal et al., 2016; Peng et al., 2016).”
MSˆ2: Multi-Document Summarization of Medical Studies (2021, methodology): “The Chemical-Disease Relations challenge (Wei et al., 2015) offers data for some of the PICO classes and a related relation extraction task, as does the i2b2 2010 disease-relation task (Uzuner et al., 2011).”