What do we know and when do we know it?

By Anthony Nicholls. 2008

219 citations recorded by the source corpus.

Two essential aspects of virtual screening are considered: experimental design and performance metrics. In the design of any retrospective virtual screen, choices have to be made as to the purpose of the exercise. Is the goal to compare methods? Is the interest in a particular type of target or all targets? Are we simulating a ‘real-world’ setting, or teasing out distinguishing features of a method? What are the confidence limits for the results? What should be reported in a publication? In particular, what criteria should be used to decide between different performance metrics? Comparing the field of molecular modeling to other endeavors, such as medical statistics, criminology, or computer hardware evaluation indicates some clear directions. Taken together these suggest the modeling field has a long way to go to provide effective assessment of its approaches, either to itself or to a broader audience, but that there are no technical reasons why progress cannot be made.

Subfields: Artificial Intelligence, Biophysics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biophysics

In Computer Vision and Pattern Recognition

Citations in the corpus

Structure-Based Virtual Screening: From Classical to Artificial Intelligence (2020, background): “However, Nicholls (Nicholls, 2008) say that AUC-ROC and BEDROC correlate when considering virtual screening simulations, and therefore, the ROC curve is a suﬃcient metric for performance measurements.”
Open-source platform to benchmark fingerprints for ligand-based virtual screening (2013, methodology): “Just as there is wide variety of fingerprinting algorithms, there are multiple methods for evaluating VS performance and little consensus as to which is best [9,11].”
FRED and HYBRID docking performance on standardized datasets (2012, methodology): “AUC is a global enrichment metric [5].”
Comparison of Several Molecular Docking Programs: Pose Prediction and Virtual Screening Accuracy (2009, methodology): “26 This unstable behavior occurs precisely where there is the most interest in the results of data analysis, at the earliest portion of the screened database.”