A Web Resource for Standardized Benchmark Datasets, Metrics, and Rosetta Protocols for Macromolecular Modeling and Design
By Shane Ó Conchúir, Kyle A. Barlow, Roland A. Pache, Noah Ollikainen, K. Kundert, Matthew J. O’Meara, Colin A. Smith, Tanja Kortemme. 2015
82 citations recorded by the source corpus.
The development and validation of computational macromolecular modeling and design methods depend on suitable benchmark datasets and informative metrics for comparing protocols. In addition, if a method is intended to be adopted broadly in diverse biological applications, there needs to be information on appropriate parameters for each protocol, as well as metrics describing the expected accuracy compared to experimental data. In certain disciplines, there exist established benchmarks and public resources where experts in a particular methodology are encouraged to supply their most efficient implementation of each particular benchmark. We aim to provide such a resource for protocols in macromolecular modeling and design. We present a freely accessible web resource (https://kortemmelab.ucsf.edu/benchmarks) to guide the development of protocols for protein modeling and design. The site provides benchmark datasets and metrics to compare the performance of a variety of modeling protocols using different computational sampling methods and energy functions, providing a “best practice” set of parameters for each method. Each benchmark has an associated downloadable benchmark capture archive containing the input files, analysis scripts, and tutorials for running the benchmark. The captures may be run with any suitable modeling method; we supply command lines for running the benchmarks using the Rosetta software suite. We have compiled initial benchmarks for the resource spanning three key areas: prediction of energetic effects of mutations, protein design, and protein structure prediction, each with associated state-of-the-art modeling protocols. With the help of the wider macromolecular modeling community, we hope to expand the variety of benchmarks included on the website and continue to evaluate new iterations of current methods as they become available.
Subfields: Materials Chemistry, Molecular Biology
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Materials Chemistry
- Tommi S. Jaakkola
- Regina Barzilay
- Jeffrey J. Gray
- Daisuke Kuroda
- Wouter Boomsma
- Vikas K. Garg
- John Ingraham
- J. Jeliazkov
In Molecular Biology
- Tommi S. Jaakkola
- Regina Barzilay
- Jeffrey J. Gray
- Daisuke Kuroda
- Wouter Boomsma
- Jes Frellsen
- Vikas K. Garg
- John Ingraham
Citations in the corpus
- The Rosetta all-atom energy function for macromolecular modeling and design (2017, neutral): “49,50,93 These benchmarks were chosen to test recovery of native-like structural features, ranging from individual hydrogen bond geometries to thermodynamic properties and interface conformations.”
- Generative Models for Graph-Based Protein Design (2019, result): “In the second, we also compared to a prior benchmark from members of the Rosetta community [48, 49] across 40 diverse proteins.”
- Repertoire Analysis of Antibody CDR-H3 Loops Suggests Affinity Maturation Does Not Typically Result in Rigidification (2017, methodology): “To investigate this hypothesis, we selected 10 mature antibodies from our SAbDab set with CDR-H3 loops of length 10, a length for which loop modeling performs well (50, 61).”
- Spherical convolutions and their application in molecular modelling (2017, methodology): “The former were taken from a recent publication (Conchúir et al., 2015), while the latter were calculated using the FoldX program (version 4).”