The Evaluation of Tools Used to Predict the Impact of Missense Variants Is Hindered by Two Types of Circularity

By Dominik G. Grimm, Chloé-Agathe Azencott, Fabian Aicheler, Udo Gieraths, Daniel G. MacArthur, Kaitlin E. Samocha, David N. Cooper, Peter D. Stenson, Mark J. Daly, Jordan W. Smoller and 2 more. 2015

342 citations recorded by the source corpus.

Prioritizing missense variants for further experimental investigation is a key challenge in current sequencing studies for exploring complex and Mendelian diseases. A large number of in silico tools have been employed for the task of pathogenicity prediction, including PolyPhen‐2, SIFT, FatHMM, MutationTaster‐2, MutationAssessor, Combined Annotation Dependent Depletion, LRT, phyloP, and GERP++, as well as optimized methods of combining tool scores, such as Condel and Logit. Due to the wealth of these methods, an important practical question to answer is which of these tools generalize best, that is, correctly predict the pathogenic character of new variants. We here demonstrate in a study of 10 tools on five datasets that such a comparative evaluation of these tools is hindered by two types of circularity: they arise due to (1) the same variants or (2) different variants from the same protein occurring both in the datasets used for training and for evaluation of these tools, which may lead to overly optimistic results. We show that comparative evaluations of predictors that do not address these types of circularity may erroneously conclude that circularity confounded tools are most accurate among all tools, and may even outperform optimized combinations of tools.

Subfields: Genetics, Molecular Biology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Genetics

In Molecular Biology

Citations in the corpus

Quantitative missense variant effect prediction using large-scale mutagenesis data (2017, neutral): “of predictor performance on clinical data is difficult given that many predictors are trained on or optimized to predict these data (Grimm et al., 2015).”
Accurate proteome-wide missense variant effect prediction with AlphaMissense (2023, background): “2A;seesupplementarynote)( 11 , 17 , 29 ).”
Protein design and variant prediction using autoregressive generative models (2021, background): “…only on evolutionary sequences are proving the most accurate for predicting the effect of mutations when compared to large data-sets of experimentally measured mutation effects 30,34 , and they avoid the risk of over ﬁ tting that can occur as a result of circularity in supervised methods 68 .”
ClinPred: Prediction Tool to Identify Disease-Relevant Nonsynonymous Single-Nucleotide Variants. (2018, methodology): “…database for neutral SNVs. 11 In order to prevent type II circularity that arises when all the variants in a gene are labeled either pathogenic or benign, 12 we retained only genes that contained both benign (VariSNP) and pathogenic (mutagenetix) variants, to create the MouseVariSNP test data.”
Variation Interpretation Predictors: Principles, Types, Performance, and Choice (2016, methodology): “A study of 10 prediction methods that were evaluated with five datasets [Grimm et al., 2015] revealed that the performance of methods decreases when the variations present in the methods’ training dataset were eliminated from the test data.”
Evaluation of in silico algorithms for use with ACMG/AMP clinical variant interpretation guidelines (2017, methodology): “dataset (Varibenchselected [31]) and a balanced dataset which includes equal number of pathogenic and benign”
Using deep mutational scanning to benchmark variant effect predictors and identify disease mutations (2020, neutral): “Grimm et al (2015) describe two types of data circularity that can bias the assessment of predictor accuracy.”