Recommendations for evaluation of computational methods

By Ajay N. Jain, Anthony Nicholls. 2008

336 citations recorded by the source corpus.

The field of computational chemistry, particularly as applied to drug design, has become increasingly important in terms of the practical application of predictive modeling to pharmaceutical research and development. Tools for exploiting protein structures or sets of ligands known to bind particular targets can be used for binding-mode prediction, virtual screening, and prediction of activity. A serious weakness within the field is a lack of standards with respect to quantitative evaluation of methods, data set preparation, and data set sharing. Our goal should be to report new methods or comparative evaluations of methods in a manner that supports decision making for practical applications. Here we propose a modest beginning, with recommendations for requirements on statistical reporting, requirements for data sharing, and best practices for benchmark preparation and usage.

Subfields: Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

Citations in the corpus

Molecular Docking and Structure-Based Drug Design Strategies (2015, background): “In short, the preparation routine consists of adding hydrogen atoms, eliminating water molecules (with the exception of those mediating important interactions), specifying the correct protonation and tautomerization states of the binding site residues, and calculating partial charges [172].”
Molecular graph convolutions: moving beyond fingerprints (2016, methodology): “Biology takes place in a three-dimensional world, and can be more sensitive to shape, electrostatics, quantum eﬀects, and other properties that emerge from the molecular graph than the structure of the graph itself (see, e.g. , Nicholls and Grant (2005); Böhm et al. (2004), and related literature on”
Massively Multitask Networks for Drug Discovery (2015, methodology): “For completeness, we include in the Appendix an alternative metric called “enrichment” that is widely used in the cheminformatics literature (Jain & Nicholls, 2008).”
Open-source platform to benchmark fingerprints for ligand-based virtual screening (2013, background): “For example, the number and diversity of protein targets, actives and inactives has to be large enough [10], and an error estimate of the performance should be obtained by boot-strapping, i.”
Comparison of Several Molecular Docking Programs: Pose Prediction and Virtual Screening Accuracy (2009, background): “A simple way to avoid this kind of bias is to plot the fraction of complexes that have poses below a given RMSD 25,28 (Figure 1).”
Advances and Challenges in Protein-Ligand Docking (2010, background): “Finally, how to evaluate different docking programs and scoring functions is another active area [153–157].”
ROCS-derived features for virtual screening (2016, background): “ROC enrichment (Ex) measures performance early in the ranked list of library molecules, calculated as the ratio of the true positive rate (TPR) and the false positive rate (FPR) at a specific FPR [Jain and Nicholls, 2008].”
Virtual screening: an endless staircase? (2010, methodology): “The issue of proper benchmarking and scrutiny of methods has also been realize”