A Comprehensive Benchmark of Kernel Methods to Extract Protein–Protein Interactions from Literature

By Domonkos Tikk, Philippe E. Thomas, Peter Palaga, Jörg Hakenberg, Ulf Leser. 2010

206 citations recorded by the source corpus.

The most important way of conveying new findings in biomedical research is scientific publication. Extraction of protein–protein interactions (PPIs) reported in scientific publications is one of the core topics of text mining in the life sciences. Recently, a new class of such methods has been proposed - convolution kernels that identify PPIs using deep parses of sentences. However, comparing published results of different PPI extraction methods is impossible due to the use of different evaluation corpora, different evaluation metrics, different tuning procedures, etc. In this paper, we study whether the reported performance metrics are robust across different corpora and learning settings and whether the use of deep parsing actually leads to an increase in extraction quality. Our ultimate goal is to identify the one method that performs best in real-life scenarios, where information extraction is performed on unseen text and not on specifically prepared evaluation data. We performed a comprehensive benchmarking of nine different methods for PPI extraction that use convolution kernels on rich linguistic information. Methods were evaluated on five different public corpora using cross-validation, cross-learning, and cross-corpus evaluation. Our study confirms that kernels using dependency trees generally outperform kernels based on syntax trees. However, our study also shows that only the best kernel methods can compete with a simple rule-based approach when the evaluation prevents information leakage between training and test corpora. Our results further reveal that the F-score of many approaches drops significantly if no corpus-specific parameter optimization is applied and that methods reaching a good AUC score often perform much worse in terms of F-score. We conclude that for most kernels no sensible estimation of PPI extraction performance on new text is possible, given the current heterogeneity in evaluation data. Nevertheless, our study shows that three kernels are clearly superior to the other methods.

Subfields: Artificial Intelligence, Molecular Biology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

Citations in the corpus

Deep learning for extracting protein-protein interactions from biomedical literature (2017, methodology): “While PPI relations can span over sentences and even cross documents, current works mostly focus on PPI in individual sentences (Pyysalo et al., 2008; Tikk et al., 2010).”
Bridging semantics and syntax with graph algorithms - state-of-the-art of extracting biomedical relations (2017, methodology): “They used the resulting dependency graphs with two graph kernels including the all paths graph (APG) kernel [20] and kBSPS [99], which produced only moderate performance.”
Extraction of potential adverse drug events from medical case reports (2012, methodology): “Amongst different kernels available, the shallow linguistic kernel was thoroughly used since it has been widely applied and has shown success during similar relation extraction tasks [17].”
Bridging semantics and syntax with graph algorithms —state-of-the-art of extracting biomedical relations (2015, background): “Several follow up studies to BioCreative-II.5 concerned the usage of kernels in PPI extraction [102,103], and categorized kernels into the following categories: 1) shallow linguistic (SL) kernels [104]; 2) constituent parse tree based kernels, including subtree (ST) [105], subset tree (SST) [106]…”
Improving chemical disease relation extraction with rich features and weakly labeled data (2016, methodology): “SVM-based systems using rich features have been previously studied in biomedical relation extraction [5, 50, 51].”