Critical Assessment of QSAR Models of Environmental Toxicity against Tetrahymena pyriformis: Focusing on Applicability Domain and Overfitting by Variable Selection

By Igor V. Tetko, Iurii Sushko, Anil Kumar Pandey, Hao Zhu, Alexander Tropsha, Ester Papa, Tomas Öberg, Roberto Todeschini, Denis Fourches, Alexandre Varnek. Journal of Chemical Information and Modeling, 2008

393 citations recorded by the source corpus.

The estimation of the accuracy of predictions is a critical problem in QSAR modeling. The "distance to model" can be defined as a metric that defines the similarity between the training set molecules and the test set compound for the given property in the context of a specific model. It could be expressed in many different ways, e.g., using Tanimoto coefficient, leverage, correlation in space of models, etc. In this paper we have used mixtures of Gaussian distributions as well as statistical tests to evaluate six types of distances to models with respect to their ability to discriminate compounds with small and large prediction errors. The analysis was performed for twelve QSAR models of aqueous toxicity against T. pyriformis obtained with different machine-learning methods and various types of descriptors. The distances to model based on standard deviation of predicted toxicity calculated from the ensemble of models afforded the best results. This distance also successfully discriminated molecules with low and large prediction errors for a mechanism-based model developed using log P and the Maximum Acceptor Superdelocalizability descriptors. Thus, the distance to model metric could also be used to augment mechanistic QSAR models by estimating their prediction errors. Moreover, the accuracy of prediction is mainly determined by the training set data distribution in the chemistry and activity spaces but not by QSAR approaches used to develop the models. We have shown that incorrect validation of a model may result in the wrong estimation of its performance and suggested how this problem could be circumvented. The toxicity of 3182 and 48774 molecules from the EPA High Production Volume (HPV) Challenge Program and EINECS (European chemical Substances Information System), respectively, was predicted, and the accuracy of prediction was estimated. The developed models are available online at http://www.qspr.org site.

Subfields: Computational Theory and Mathematics, Materials Chemistry, Spectroscopy

View Paper: https://www.semanticscholar.org/paper/20e8c27ef7ef6ec0e8c1fb16b131df01b8eb2ad0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Theory and Mathematics

In Materials Chemistry

In Spectroscopy

Citations in the corpus

Reliable estimation of prediction errors for QSAR models under model uncertainty using double cross-validation (2014, neutral): “Although many successful applications of double cross-validation have been published in recent years [7,9,10,12,14,16,18,19,51-56], there is still some reluctance to use double cross-validation.”
Calculation of molecular lipophilicity: state of the art and comparison of methods on more than 96000 compounds (2009, neutral): “pyriformis in a benchmarking study of 12 models and 6 types of applicability domain methods.(167) Figure 4 shows that prediction accuracy of the analyzed models gradually decreases with the StdDev.”
Extended Functional Groups (EFG): An Efficient Set for Chemical Characterization and Structure-Activity Relationship Studies of Chemical Compounds (2015, neutral): “Pyriformis [17] Estate 644 ASNN 0.”
The development of models to predict melting and pyrolysis point data associated with several hundred thousand compounds mined from PATENTS (2016, methodology): “In our previous work [11] we found that ASNN [44] and SVM [45] methods provided significantly higher accuracy of MP predictions compared to other tested methods while the accuracy of models developed with both methods was similar.”
Online chemical modeling environment (OCHEM): web platform for data storage, model development and publishing of chemical information (2011, methodology): “Pyriformis bacteria [29] included a collection of 1,093 experimental measurements used for the model, which is available at http://www.”
Comparison of Different Approaches to Define the Applicability Domain of QSAR Models (2012, neutral): “5 as default value [26].”
A renaissance of neural networks in drug discovery (2016, background): “Benchmarking of different definitions of the applicability domain identified STD as the best measure of prediction reliability of molecules from regression studies [65].”
Introduction to the BioChemical Library (BCL): An Application-Based Open-Source Toolkit for Integrated Cheminformatics and Machine Learning in Computer-Aided Drug Discovery (2022, background): “This is a good example that internal randomized cross-validation on a small training set is not an accurate predictor of external test set performance unless the external test set is within a similar domain of applicability (Tetko et al., 2008; Sheridan, 2012).”