Evaluation of model quality predictions in CASP9

By Andriy Kryshtafovych, Krzysztof Fidelis, Anna Tramontano. 2011

90 citations recorded by the source corpus.

CASP has been assessing the state of the art in the a priori estimation of accuracy of protein structure prediction since 2006. The inclusion of model quality assessment category in CASP contributed to a rapid development of methods in this area. In the last experiment, 46 quality assessment groups tested their approaches to estimate the accuracy of protein models as a whole and/or on a per‐residue basis. We assessed the performance of these methods predominantly on the basis of the correlation between the predicted and observed quality of the models on both global and local scales. The ability of the methods to identify the models closest to the best one, to differentiate between good and bad models, and to identify well modeled regions was also analyzed. Our evaluations demonstrate that even though global quality assessment methods seem to approach perfection point (weighted average per‐target Pearson's correlation coefficients are as high as 0.97 for the best groups), there is still room for improvement. First, all top‐performing methods use consensus approaches to generate quality estimates, and this strategy has its own limitations. Second, the methods that are based on the analysis of individual models lag far behind clustering techniques and need a boost in performance. The methods for estimating per‐residue accuracy of models are less accurate than global quality assessment methods, with an average weighted per‐model correlation coefficient in the range of 0.63–0.72 for the best 10 groups. Proteins 2011; © 2011 Wiley‐Liss, Inc.

Subfields: Molecular Biology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Molecular Biology

Citations in the corpus

Assessment of model accuracy estimations in CASP12 (2017, methodology): “In this section, we assess the effectiveness of EMA methods to assign overall accuracy score to a model by evaluating their ability to (1) find the best model amongst many others, (2) reproduce model-target similarity scores, and (3) discriminate between good and bad models.”
Methods of model accuracy estimation can help selecting the best models from decoy sets: assessment of model accuracy estimations in CASP11 (2015, background): “Estimation of protein model accuracy is of great practical interest in structural biology1–3 and remains an active area in bioinformatics research.”
The ModFOLD4 server for the quality assessment of 3D protein models (2013, methodology): “However, the CASP8 and CASP9 assessments have been criticized for disadvantaging singlemodel methods and quasi-single-model methods in comparison with clustering methods, owing to availability of large sets of models (17).”
Improved model quality assessment using ProQ2 (2012, methodology): “On the other hand, in terms of accuracy the consensus and hybrid methods outperform the single methods, in particular in benchmarks such as CASP [25] with access tomany alternative models for all different targets.”
Improved de novo structure prediction in CASP11 by incorporating coevolution information into Rosetta (2016, methodology): “31,32 Simply re-ranking the five submitted Robetta models using the MQA method, ProQ2,33 significantly improves model 1 accuracy for the FM domains (Fig.”
The utility of comparative models and the local model quality for protein crystal structure determination by Molecular Replacement (2012, methodology): “In the CASP9 experiment, MetaMQAP and MetaM-QAPclust were shown to belong to the groups of best performing “ true MQAPs ” and “ clustering MQAPs ” , respectively [28].”
OpenStructure: an integrated software framework for computational structural biology (2013, unknown): “In fact, for many applications, methods based on evolutionary information considerably outperform physicsbased approaches (Kryshtafovych et al., 2011; Mariani, Kiefer et al., 2011).”
Assessment of template based protein structure predictions in CASP10 (2014, methodology): “1,16 In particular, we followed the general procedure using standard measures for TBM assessment outlined by Cozzetto et al .,15 and considerations of structure prediction accuracy metrics and statistical comparison tests outlined by Mariani et al .1 for the CASP9 TBM assessment.”