A study of quality measures for protein threading models

By S. Cristobal, Adam T. Zemla, Daniel Fischer, Leszek Rychlewski, Arne Elofsson. 2001

222 citations recorded by the source corpus.

BackgroundPrediction of protein structures is one of the fundamental challenges in biology today. To fully understand how well different prediction methods perform, it is necessary to use measures that evaluate their performance. Every two years, starting in 1994, the CASP (Critical Assessment of protein Structure Prediction) process has been organized to evaluate the ability of different predictors to blindly predict the structure of proteins. To capture different features of the models, several measures have been developed during the CASP processes. However, these measures have not been examined in detail before. In an attempt to develop fully automatic measures that can be used in CASP, as well as in other type of benchmarking experiments, we have compared twenty-one measures. These measures include the measures used in CASP3 and CASP2 as well as have measures introduced later. We have studied their ability to distinguish between the better and worse models submitted to CASP3 and the correlation between them.ResultsUsing a small set of 1340 models for 23 different targets we show that most methods correlate with each other. Most pairs of measures show a correlation coefficient of about 0.5. The correlation is slightly higher for measures of similar types. We found that a significant problem when developing automatic measures is how to deal with proteins of different length. Also the comparisons between different measures is complicated as many measures are dependent on the size of the target. We show that the manual assessment can be reproduced to about 70% using automatic measures. Alignment independent measures, detects slightly more of the models with the correct fold, while alignment dependent measures agree better when selecting the best models for each target. Finally we show that using automatic measures would, to a large extent, reproduce the assessors ranking of the predictors at CASP3.ConclusionsWe show that given a sufficient number of targets the manual and automatic measures would have given almost identical results at CASP3. If the intent is to reproduce the type of scoring done by the manual assessor in in CASP3, the best approach might be to use a combination of alignment independent and alignment dependent measures, as used in several recent studies.

Subfields: Molecular Biology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Molecular Biology

Citations in the corpus

DockQ: A Quality Measure for Protein-Protein Docking Models (2016, methodology): “Highlighting, yet again, the importance of using continuous measures like DockQ or IS-score, which do not exhibit the same problems.”
Using multiple templates to improve quality of homology models in automated homology modeling (2008, background): “Many other evaluation functions exist, such as LG score (Cristobal et al. 2001) and MaxSub (Siew et al. 2000).”
MUSTER: Improving protein sequence profile–profile alignments by using multiple sources of structure information (2008, background): “14,30,57,58 There are a few protein pairs in the PROSUP databases which do not have similar topology.”
3DCoffee: combining protein sequences and structures within multiple sequence alignments. (2004, unknown): “Sequence–structure comparisons (or threading) can be achieved using two categories of methods.(29,30) One may use techniques inspired from molecular replacement to check whether a sequence is compatible with a 3D fold,(31) or sophisticated DP where the algorithm analyses the 3D-structure to determin”
Identification of correct regions in protein models using structural, alignment, and consensus information (2006, methodology): “2000), LGscore (Cristobal et al. 2001), and TM-score (Zhang and Skolnick 2004).”
Methods for estimation of model accuracy in CASP12 (2017, methodology): “After superposition, the “S-score” is calculated for each residue in the model.(23) The average S-score for all residues and pairs of models is then used to calculate the final Pcons score.”
Pcons5: combining consensus, structural evaluation and fold recognition scores (2005, methodology): “In more detail, the first cutoff, was set to the score for which 50% of all models had an LGscore>1.5 and the second, to the score For scores in between the cutoff values a linear fit is used, e.g. a score of 10 −5 would yield a scaled score of 0.81 on the " good " scale and 0 on the " very good " (”
Can correct protein models be identified? (2003, methodology): “Exactly how to define what conformations are native-like is not trivial, but several different measures have been developed (for review, see Cristobal et al. 2001).”