Methods of model accuracy estimation can help selecting the best models from decoy sets: assessment of model accuracy estimations in CASP11

By Andriy Kryshtafovych, Alessandro Barbato, Bohdan Monastyrskyy, Krzysztof Fidelis, Torsten Schwede, Anna Tramontano. 2015

77 citations recorded by the source corpus.

The article presents assessment of the model accuracy estimation methods participating in CASP11. The results of the assessment are expected to be useful to both—developers of the methods and users who way too often are presented with structural models without annotations of accuracy. The main emphasis is placed on the ability of techniques to identify the best models from among several available. Bivariate descriptive statistics and ROC analysis are used to additionally assess the overall correctness of the predicted model accuracy scores, the correlation between the predicted and observed accuracy of models, the effectiveness in distinguishing between good and bad models, the ability to discriminate between reliable and unreliable regions in models, and the accuracy of the coordinate error self‐estimates. A rigid‐body measure (GDT_TS) and three local‐structure‐based scores (LDDT, CADaa, and SphereGrinder) are used as reference measures for evaluating methods' performance. Consensus methods, taking advantage of the availability of several models for the same target protein, perform well on the majority of tasks. Methods that predict accuracy on the basis of a single model perform comparably to consensus methods in picking the best models and in the estimation of how accurate is the local structure. More groups than in previous experiments submitted reasonable error estimates of their own models, most likely in response to a recommendation from CASP and the increasing demand from users. Proteins 2016; 84(Suppl 1):349–369. © 2015 Wiley Periodicals, Inc.

Subfields: Materials Chemistry, Molecular Biology, Spectroscopy, Evaluation & Benchmarks, Computational Biology & Bioinformatics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Materials Chemistry

In Molecular Biology

In Spectroscopy

In Evaluation & Benchmarks

In Computational Biology & Bioinformatics

Citations in the corpus

The SWISS-MODEL Repository—new features and functionality (2016, background): “(31) for a recent benchmark of various tools).”
Critical Assessment of Methods of Protein Structure Prediction (CASP) – Progress and New directions in Round XI (2016, methodology): “45 This focus has resulted in a steady improvement in the capabilities of methods in this area46–49 and has contributed to the development of new methods and improvement of the existing methods that are capable of estimating accuracy on the basis of a single model (as opposed to methods that…”
ProQ3D: improved model quality assessments using deep learning (2016, methodology): “The ProQ methods have since their introduction been the best single-model based quality assessors in CASP (Kryshtafovych et al., 2016).”
Assessment of model accuracy estimations in CASP12 (2017, methodology): “The effectiveness of 24 CASP12 local model accuracy estimators is evaluated by verifying how well these methods (1) assign correct distance errors at the residue level, and (2) discriminate between reliable and unreliable regions in a model.”
Deep convolutional networks for quality assessment of protein folds (2018, methodology): “In this work we explore the application of deep learning to the problem of “model quality assessment” (MQA), also called “estimation of model accuracy” (EMA) [3].”
Methods for estimation of model accuracy in CASP12 (2017, methodology): “Since CASP11, however, they perform on par or even better than the consensus methods in some of the tasks.(8) The first successful attempt of model quality estimation, in the context of CASP, was when the first meta-predictor, CAFASP-CONSENSUS, was introduced in CASP4.”
ProQ3: Improved model quality assessments using Rosetta energy terms (2016, methodology): “However, it has lately been realised that these methods have their limitations [9].”
Evaluation of the template-based modeling in CASP12 (2017, methodology): “The ASE measure was previously used to score model accuracy estimates, 14 and is used here to emphasize the importance of predicting atomic level errors.”
CASP11 Statistics and the Prediction Center Evaluation System (2016, background): “22 The residue–residue contact predictions in CASP11 showed exciting developments and were evaluated with a number of measures, including precision, recall, Xd-scores, the Matthews correlation coefficients, as well as precision-recall curves.”