Overview of the protein-protein interaction annotation extraction task of BioCreative II

By Martin Krallinger, F. Leitner, Carlos Rodríguez-Penagos, Alfonso Valencia. Genome Biology, 2008

309 citations recorded by the source corpus.

Background:The biomedical literature is the primary information source for manual protein-protein interaction annotations. Text-mining systems have been implemented to extract binary protein interactions from articles, but a comprehensive comparison between the different techniques as well as with manual curation was missing.Results:We designed a community challenge, the BioCreative II protein-protein interaction (PPI) task, based on the main steps of a manual protein interaction annotation workflow. It was structured into four distinct subtasks related to: (a) detection of protein interaction-relevant articles; (b) extraction and normalization of protein interaction pairs; (c) retrieval of the interaction detection methods used; and (d) retrieval of actual text passages that provide evidence for protein interactions. A total of 26 teams submitted runs for at least one of the proposed subtasks. In the interaction article detection subtask, the top scoring team reached an F-score of 0.78. In the interaction pair extraction and mapping to SwissProt, a precision of 0.37 (with recall of 0.33) was obtained. For associating articles with an experimental interaction detection method, an F-score of 0.65 was achieved. As for the retrieval of the PPI passages best summarizing a given protein interaction in full-text articles, 19% of the submissions returned by one of the runs corresponded to curator-selected sentences. Curators extracted only the passages that best summarized a given interaction, implying that many of the automatically extracted ones could contain interaction information but did not correspond to the most informative sentences.Conclusion:The BioCreative II PPI task is the first attempt to compare the performance of text-mining tools specific for each of the basic steps of the PPI extraction pipeline. The challenges identified range from problems in full-text format conversion of articles to difficulties in detecting interactor protein pairs and then linking them to their database records. Some limitations were also encountered when using a single (and possibly incomplete) reference database for protein normalization or when limiting search for interactor proteins to co-occurrence within a single sentence, when a mention might span neighboring sentences. Finally, distinguishing between novel, experimentally verified interactions (annotation relevant) and previously known interactions adds additional complexity to these tasks.

Subfields: Artificial Intelligence, Molecular Biology

View Paper: https://www.semanticscholar.org/paper/78d0078bfc398008175faa2066ba833bca6883d7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhiyong Lu
2. Yuan Luo
3. Robert Leaman
4. Goran Nenadic
5. Peter Szolovits
6. Ozlem Uzuner
7. Ulf Leser
8. K. Cohen

In Molecular Biology

1. Zhiyong Lu
2. Yuan Luo
3. Robert Leaman
4. Ozlem Uzuner
5. Peter Szolovits
6. Ulf Leser
7. K. Cohen
8. Goran Nenadic

Citations in the corpus

• LINNAEUS: A species name identification system for biomedical literature (2010, background): “In recent text-mining challenges such as the identification of protein-protein-interactions at BioCreative II [8] or bio-molecular event extraction at the BioNLP shared task [9], some groups considered species identification and normalization an essential sub-task [10].”
• Bridging semantics and syntax with graph algorithms —state-of-the-art of extracting biomedical relations (2015, unknown): “Among the participants of the protein-protein interaction task of BioCreative-II [97], most systems used co-occurrence statistics, pattern templates and shallow linguistic features (e.g., context words and part-of-speech tags), with either statistical machine learning or rule-based systems.”
• Bridging semantics and syntax with graph algorithms - state-of-the-art of extracting biomedical relations (2017, methodology): “Among the participants of the PPI task of BioCreative-II [132], most systems used co-occurrence statistics, pattern templates and shallow linguistic features (e.”
• Simple tricks for improving pattern-based information extraction from the biomedical literature (2010, background): “While earlier shared tasks (such as BioCreative II, [20]) have dealt with general PPI, the BioNLP task focused on a set of more specific molecular events, such as gene expression, protein catabolism, or positive regulation and negative regulation.”
• Deep learning for extracting protein-protein interactions from biomedical literature (2017, background): “One of the most important tasks is to extract protein-protein interaction relations (Krallinger et al., 2008).”
• A Comprehensive Benchmark of Kernel Methods to Extract Protein–Protein Interactions from Literature (2010, methodology): “With 78 hand-crafted templates, they achieve an F-score of 29% on the BioCreative 2 IPS test set [43], which was the best at the time of the competition.”
• Construction of an annotated corpus to support biomedical information extraction (2009, background): “Within the field of bio-IE, evaluations such as the LLL05 challenge [13] and BioCreative II [14] have focussed attention on the recognition of protein-protein interaction (PPI) events from the literature.”
• The SPECIES and ORGANISMS Resources for Fast and Accurate Identification of Taxonomic Names in Text (2013, background): “Accurately identifying the taxa mentioned in a document is an important task, both in its own right to classify and retrieve documents, as a prerequisite for resolving inter-species ambiguities when identifying gene and protein names [4,5], and as a component in natural language systems [6].”
• PubMed and beyond: a survey of web tools for searching biomedical literature (2011, background): “Problems in these areas have received the most attention in the text mining community (68,69).”
• Overview of BioCreative II gene normalization (2008, methodology): “These simplifications may help to explain discrepancies in performance on the 'idealized' GN task described here and the performance of systems on a more realistic protein normalization task performed in the context of the PPI task [7].”