Overview of BioCreAtIvE: critical assessment of information extraction for biology

By Lynette Hirschman, Alexander S. Yeh, C. Blaschke, Alfonso Valencia. BMC Bioinformatics, 2005

587 citations recorded by the source corpus.

BackgroundThe goal of the first BioCreAtIvE challenge (Critical Assessment of Information Extraction in Biology) was to provide a set of common evaluation tasks to assess the state of the art for text mining applied to biological problems. The results were presented in a workshop held in Granada, Spain March 28–31, 2004. The articles collected in this BMC Bioinformatics supplement entitled "A critical assessment of text mining methods in molecular biology" describe the BioCreAtIvE tasks, systems, results and their independent evaluation.ResultsBioCreAtIvE focused on two tasks. The first dealt with extraction of gene or protein names from text, and their mapping into standardized gene identifiers for three model organism databases (fly, mouse, yeast). The second task addressed issues of functional annotation, requiring systems to identify specific text passages that supported Gene Ontology annotations for specific proteins, given full text articles.ConclusionThe first BioCreAtIvE assessment achieved a high level of international participation (27 groups from 10 countries). The assessment provided state-of-the-art performance results for a basic task (gene name finding and normalization), where the best systems achieved a balanced 80% precision / recall or better, which potentially makes them suitable for real applications in biology. The results for the advanced task (functional annotation from free text) were significantly lower, demonstrating the current limitations of text-mining approaches where knowledge extrapolation and interpretation are required. In addition, an important contribution of BioCreAtIvE has been the creation and release of training and test data sets for both tasks. There are 22 articles in this special issue, including six that provide analyses of results or data quality for the data sets, including a novel inter-annotator consistency assessment for the test set used in task 2.

Subfields: Molecular Biology

View Paper: https://www.semanticscholar.org/paper/0f103bf6fd9b7b91aa692e3f3ff7d370593a8900

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Molecular Biology

Citations in the corpus

Unsupervised biomedical named entity recognition: Experiments with clinical and biological texts (2013, unknown): “In the ﬁrst BioCreAtIvE challenge [47], gene mention identiﬁcation was the ﬁrst subtask of task1 [9].”
The BioGRID interaction database: 2019 update (2018, background): “BioGRID is a longstanding participant in the BioCreative consortium that aims to develop and benchmark biomedical text-mining approaches (40).”
Text-mining and information-retrieval services for molecular biology (2005, methodology): “To identify biological entities such as genes, proteins and drugs automatically and unambiguously within free text, over 50 information-extraction and text-mining tools have recently been implemented, and two community-wide evaluations have been carried out [19,20].”
DNorm: disease name normalization with pairwise learning to rank (2013, background): “Biomedical named entity recognition (NER) research has received increased attention recently, partly owing to BioCreative (Hirschman et al. , 2005b) and BioNLP (Kim et al. , 2009) challenges on recognition of genes, proteins and biological events in the scientific literature, as well as TREC…”
Biomedical language processing: what's beyond PubMed? (2006, background): “Within a species, a single symbol may name multiple genes— Hirschman et al. (2005b) report that 3.6% of all Drosophila identifiers map to multiple FlyBase entries.”
Accelerated training of conditional random fields with stochastic gradient methods (2006, methodology): “Namedentity recognition aims to identify and classify technical terms in a given domain (here: molecular biology) that refer to concepts of interest to domain experts (Kim et al., 2004).”
MINT, the molecular interaction database: 2012 update (2011, methodology): “Over the past years MINT has participated in the BioCreAtIvE (Critical Assessment of Information Extraction systems in Biology) challenge (13) by providing manually curated datasets for the assessment exercise.”
A Sequential Dual Method for Structural SVMs (2011, methodology): “Five data sets, Part-of-Speech (POS) [18], Wall Street Journal POS (WSJPOS) [16], BioNLP [14], BioCreative [6] and CoNLL [21], were used in our experiments.”
Frontiers of biomedical text mining: current progress (2007, background): “Specifically, the track has engaged in a concerted effort to collect actual questions from working scientists with a broad range of backgrounds and from a wide range of working environments [119, 120].”
Text mining and its potential applications in systems biology. (2006, unknown): “BioCreAtIvE (2004) [ 56 ] (http://biocreative.sourceforge.net/) BioNLP, JNLPBA (http://www-tsujii.is.s.u-tokyo.ac.jp/GENIA/”
Constructing Datasets for Multi-hop Reading Comprehension Across Documents (2017, methodology): “The promise of applying NLP methods to cope with this increase has led to research efforts in IE (Hirschman et al., 2005; Kim et al., 2011) and QA for biomedical text (Hersh et al., 2007; Nentidis et al., 2017).”
Bidirectional RNN for Medical Event Detection in Electronic Health Records (2016, methodology): “Many named entity recognition systems in the biomedical domain have been driven by the Shared tasks of BioNLP (Kim et al., 2009), BioCreAtivE (Hirschman et al., 2005) i2b2 shared NLP tasks (Li et al., 2009) and ShARe/CLEF evaluation tasks (Pradhan et al., 2014”
Computational tools for prioritizing candidate genes: boosting disease gene discovery (2012, background): “There is a need for a large-scale community effort — similar in spirit to the CASP 107,108 , BioCreative 109,110 , CAMDA 111,112 or DREAM 113 competitions — in which multiple tools can be compared across common prospective benchmarks that have been designed by the community.”
Extracting interactions between proteins from the literature (2008, unknown): “BioCreAtIvE (Critical Assessment of Information Extraction systems in Biology) [93](3) began in 2004 and provided two common evaluation tasks to assess the state of the art methods for text mining applied to biological problems.”
The Gene Ontology (GO) project in 2006 (2005, methodology): “For example, GO terms have been used in the Textpresso text mining system and in the BioCreAtIvE text mining assessment (13,27,28).”