CASP11 Statistics and the Prediction Center Evaluation System

By Andriy Kryshtafovych, Bohdan Monastyrskyy, Krzysztof Fidelis. 2016

44 citations recorded by the source corpus.

We outline the role of the Protein Structure Prediction Center (predictioncenter.org) in conducting the CASP11 and CASP ROLL experiments, discuss the experiment statistics, and provide an overview of the present CASP infrastructure. The biggest changes compared to the previous CASPs are the implementation of the evaluation system incorporating practically all evaluation measures, statistical tests, and visualization tools historically used by the CASP assessors, the expansion of the infrastructure to incorporate new categories of contact‐assisted and multimeric predictions, and the redesign of the assessors' web‐workspace enabling assessments based on multiple measures for different group categories and target sets. Proteins 2016; 84(Suppl 1):15–19. © 2016 Wiley Periodicals, Inc.

Subfields: Materials Chemistry, Molecular Biology, Spectroscopy

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Materials Chemistry

In Molecular Biology

In Spectroscopy

Citations in the corpus

End-to-end differentiable learning of protein structure (2018, methodology): “On all CASPs, RGNs had the best performance, even compared to servers that use coevolution data (in CASP 11 (Kryshtafovych et al., 2016; Ovchinnikov et al., 2016) and CASP 12 (Schaarschmidt et al.”
High‐accuracy protein structure prediction in CASP14 (2021, background): “14 There, the different targets submitted are divided into evaluation units (or domains), each classified as “ template-based easy ” (TBM-Easy), “ template-based hard ” (TBM-Hard), “ hybrid template-based/free modeling ” (TBM/FM), or “ free-modeling ” (FM) targets, depending on whether good…”
A further leap of improvement in tertiary structure prediction in CASP13 prompts new routes for future assessments (2019, methodology): “9,10 GDTTS ranges between 0 and 100 with scores below Finally, the TM 9 score (defined from 0 to 1, higher is better) combines the distances between all residues of the aligned region in the target and model, weighting them by a reference distance calibrated from the target's length instead of consi”
Assessment of hard target modeling in CASP12 reveals an emerging role of alignment‐based contact prediction methods (2018, methodology): “8 Besides the group rankings described above, we additionally pres-ent the number of best models submitted by each group according to visual inspection (Table 2) and a summary of the visual assessment guided by the 6 scores on all target EUs (Table S2).”
Topology Evaluation of Models for Difficult Targets in the 14th Round of the Critical Assessment of Protein Structure Prediction (CASP14) (2021, methodology): “The model scores were combined by the Prediction Center using SumZ> (cid:1) 2 as previously described with the formula 1*GDT_TS + _1*QCS + 0.1* MolProbity.”
Evaluation of model refinement in CASP13 (2019, methodology): “The accuracy self-assessment (ASE) measure differs qualitatively in measuring not the accuracy of the model but rather the accuracy of the modelers' estimates of local coordinate error.(10) Presumably because the accuracy of error estimates has not been evaluated for refinement models in previous ro”
CoinFold: a web server for protein contact prediction and contact-assisted protein folding (2016, methodology): “We tested CoinFold using 228 CASP test proteins (123 CASP10 plus 105 CASP11 targets) (25), and 47 CAMEO hard test proteins (from 2015-08-01 to 2015-09-12) (26).”
Definition and classification of evaluation units for tertiary structure prediction in CASP12 facilitated through semi‐automated metrics (2018, methodology): “used in this work were performed by the Prediction Center; thus they will be readily available in future CASP rounds.(2)”
Assessment of the model refinement category in CASP12 (2017, methodology): “34,36 Here, in line with previous CASP refinement experiments, we based our analysis on the following metrics: the Root Mean Square Deviation of the C a atoms (RMSD), the Global Distance Test 33,35 (GDT_TS and GDT_HA), SphereGrinder 36 (SphGr), the Local Distance Difference Test (LDDT), 37 and the…”
Evaluation of the template-based modeling in CASP12 (2017, background): “Typically, submitted models are automatically evaluated at the Prediction Center 5 – 9 and then the results are interpreted by independent assessors.”
Target Classification in the 14th Round of the Critical Assessment of Protein Structure Prediction (CASP14). (2021, methodology): “The Prediction Center provides a number of metrics that can be used to help classify templates.(11) We chose the same metrics as in previous rounds of CASP(5,16) to assign CASP14 target EUs to evaluation categories.”
Evaluation of template-based modeling in CASP13 (2019, methodology): “Most of these metrics are computed, collated, and analyzed by the Prediction Center (http://predictioncenter. org), 22 making them much more convenient for assessors and others.”