Introducing “best single template” models as reference baseline for the Continuous Automated Model Evaluation (CAMEO)

By J. Haas, Rafal Gumienny, Alessandro Barbato, F. Ackermann, Gerardo Tauriello, Martino Bertoni, Gabriel Studer, Anna Smolinski, Torsten Schwede. 2019

34 citations recorded by the source corpus.

Critical blind assessment of structure prediction techniques is crucial for the scientific community to establish the state of the art, identify bottlenecks, and guide future developments. In Critical Assessment of Techniques in Structure Prediction (CASP), human experts assess the performance of participating methods in relation to the difficulty of the prediction task in a biennial experiment on approximately 100 targets. Yet, the development of automated computational modeling methods requires more frequent evaluation cycles and larger sets of data. The “Continuous Automated Model EvaluatiOn (CAMEO)” platform complements CASP by conducting fully automated blind prediction evaluations based on the weekly pre‐release of sequences of those structures, which are going to be published in the next release of the Protein Data Bank (PDB). Each week, CAMEO publishes benchmarking results for predictions corresponding to a set of about 20 targets collected during a 4‐day prediction window. CAMEO benchmarking data are generated consistently for all methods at the same point in time, enabling developers to cross‐validate their method's performance, and referring to their results in publications. Many successful participants of CASP have used CAMEO—either by directly benchmarking their methods within the system or by comparing their own performance to CAMEO reference data. CAMEO offers a variety of scores reflecting different aspects of structure modeling, for example, binding site accuracy, homo‐oligomer interface quality, or accuracy of local model confidence estimates. By introducing the "bestSingleTemplate" method based on structure superpositions as a reference for the accuracy of 3D modeling predictions, CAMEO facilitates objective comparison of techniques and fosters the development of advanced methods.

Subfields: Materials Chemistry, Molecular Biology, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Materials Chemistry

1. Minkyung Baek
2. Ivan V. Anishchenko
3. A. Kang
4. Frank DiMaio
5. David Baker
6. Sergey Ovchinnikov
7. Hahnbeom Park
8. David E. Kim

In Molecular Biology

1. Minkyung Baek
2. Ivan V. Anishchenko
3. Frank DiMaio
4. A. Kang
5. Jue Wang
6. Ryan McHugh
7. A. Bera
8. David Baker

In Evaluation & Benchmarks

1. Frank DiMaio
2. Ivan V. Anishchenko
3. Minkyung Baek

Citations in the corpus

• Accurate prediction of protein structures and interactions using a 3-track neural network (2021, neutral): “Fortunately,theContinuousAutomated ModelEvaluation(CAMEO)experiment( 7 ) tests structure prediction servers blindly on protein structures as they are submitted to the PDB. RoseTTAFold has been evaluated since 15 May 2021 on CAMEO; over the 69 medium and hard targets released during this time (15 May”
• Generalized Biomolecular Modeling and Design with RoseTTAFold All-Atom (2023, positive): “…blind testing of RFAA prediction performance, we enrolled an RFAA server in the blind Continuous Automated Model EvaluatiOn (CAMEO) ligand-docking evaluation, which carries out predictions on all struc- turessubmittedtothePDBeachweekwitheachenrolledserverandevaluatestheirperfor-mance( 16 – 18 ).”