Assessment of model accuracy estimations in CASP12

By Andriy Kryshtafovych, Bohdan Monastyrskyy, Krzysztof Fidelis, Torsten Schwede, Anna Tramontano. 2017

63 citations recorded by the source corpus.

The record high 42 model accuracy estimation methods were tested in CASP12. The paper presents results of the assessment of these methods in the whole‐model and per‐residue accuracy modes. Scores from four different model evaluation packages were used as the “ground truth” for assessing accuracy of methods' estimates. They include a rigid‐body score—GDT_TS, and three local‐structure based scores—LDDT, CAD and SphereGrinder. The ability of methods to identify best models from among several available, predict model's absolute accuracy score, distinguish between good and bad models, predict accuracy of the coordinate error self‐estimates, and discriminate between reliable and unreliable regions in the models was assessed. Single‐model methods advanced to the point where they are better than clustering methods in picking the best models from decoy sets. On the other hand, consensus methods, taking advantage of the availability of large number of models for the same target protein, are still better in distinguishing between good and bad models and predicting local accuracy of models. The best accuracy estimation methods were shown to perform better with respect to the frozen in time reference clustering method and the results of the best method in the corresponding class of methods from the previous CASP. Top performing single‐model methods were shown to do better than all but three CASP12 tertiary structure predictors when evaluated as model selectors.

Subfields: Computational Theory and Mathematics, Materials Chemistry, Molecular Biology, Evaluation & Benchmarks, Computational Biology & Bioinformatics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Theory and Mathematics

In Materials Chemistry

In Molecular Biology

In Evaluation & Benchmarks

In Computational Biology & Bioinformatics

Citations in the corpus

IntFOLD: an integrated web resource for high performance protein structure and function prediction (2019, positive): “Recently, the component methods have ranked among top independent servers in the Tertiary Structure (TS) prediction (5) and Estimates of Model Accuracy (EMA) categories (7), as well as ranking well in the historical categories of intrinsic disorder prediction and function prediction (26,27).”
Driven to Near-Experimental Accuracy by Refinement via Molecular Dynamics Simulations (2019, background): “56 In terms of a binary classification, identifying residues deviating more than 3.8 Å, F1, precision, and recall were 0.539, 55.9%, and 64.5% with a cutoff 2.4 Å for predictions, respectively, and the area under the curve (AUC) was 0.867.”
Methods for estimation of model accuracy in CASP12 (2017, methodology): “However, consensus-based methods such as Pcons are still superior at separating correct and incorrect models.(52) Interestingly, when”
Assessment of Protein Model Structure Accuracy Estimation in CASP13: Challenges in the Era of Deep Learning (2019, methodology): “Overall, we followed the general practice of CASP.(6) Two model accuracy measures, GDT-TS(9) for global fold accuracy and LDDT(10) for local environment accuracy, were adopted after testing several available measures.”