Statistical Evaluation of the Predictive Toxicology Challenge 2000-2001

By Hannu (TT) Toivonen, A. Srinivasan, Ross D. King, Stefan Kramer, C. Helma. 2003

221 citations recorded by the source corpus.

MOTIVATION The development of in silico models to predict chemical carcinogenesis from molecular structure would help greatly to prevent environmentally caused cancers. The Predictive Toxicology Challenge (PTC) competition was organized to test the state-of-the-art in applying machine learning to form such predictive models. RESULTS Fourteen machine learning groups generated 111 models. The use of Receiver Operating Characteristic (ROC) space allowed the models to be uniformly compared regardless of the error cost function. We developed a statistical method to test if a model performs significantly better than random in ROC space. Using this test as criteria five models performed better than random guessing at a significance level p of 0.05 (not corrected for multiple testing). Statistically the best predictor was the Viniti model for female mice, with p value below 0.002. The toxicologically most interesting models were Leuven2 for male mice, and Kwansei for female rats. These models performed well in the statistical analysis and they are in the middle of ROC space, i.e. distant from extreme cost assumptions. These predictive models were also independently judged by domain experts to be among the three most interesting, and are believed to include a small but significant amount of empirically learned toxicological knowledge. AVAILABILITY PTC details and data can be found at: http://www.predictive-toxicology.org/ptc/.

Subfields: Computational Theory and Mathematics, Immunology, Molecular Biology, Evaluation & Benchmarks, Computational Chemistry & Drug Discovery

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Theory and Mathematics

In Immunology

In Molecular Biology

In Evaluation & Benchmarks

In Computational Chemistry & Drug Discovery

Citations in the corpus

Learning Convolutional Neural Networks for Graphs (2016, background): “PTC consists of 344 chemical compounds where classes indicate carcinogenicity for male and female rats (Toivonen et al., 2003).”
Diffusion-Convolutional Neural Networks (2015, background): “PTC [9] contains 344 compounds labeled with whether they are carcinogenic in rats with 19 node features.”
Pre-training Graph Neural Networks with Kernels (2018, neutral): “We report experimental results on three datasets containing biological node-labeled graphs, namely MUTAG [Debnath et al. , 1991], PTC [Toivonen et al. , 2003], NCI1 [Wale et al. , 2008], Each graph represents a chemical compound, where nodes are labeled with the atom type, and edges represent bonds…”
A Comprehensive Survey on Graph Neural Networks (2019, neutral): “81 4 2 [26], [46], [52], [54], [57] PTC [124] 344 25.”
On Filter Size in Graph Convolutional Networks (2018, unknown): “In this setting, we eliminate MUTAG and PTC since they have a small number of examples which easily causes over-ﬁtting problems for deep learning approaches.”
Fast Neighborhood Subgraph Pairwise Distance Kernel (2010, background): “PTC: The 2000-2001 Predictive Toxicology Challenge (PTC) (Toivonen et al., 2003) was devised to stimulate the development of machine learning techniques for predictive toxicology models.”
Matching Node Embeddings for Graph Similarity (2017, background): “The PTC-MR dataset (Toivonen et al. 2003) consists of 344 organic molecules marked according to their carcinogenicity on male rats.”
Deep Graph Kernels (2015, background): “PTC [31] is a dataset of 344 chemical compounds that reports the carcinogenicity for male and female rats and it has 19 discrete labels.”
A Structural Smoothing Framework For Robust Graph Comparison (2015, background): “Smoothed kernels outperform other methods on all datasets, and the results are statistically significant on every dataset except PTC.”
Graph Self-Supervised Learning: A Survey (2021, background): “The PTC [141] and PTCMR [142] datasets contain a series of chemical compounds, while the labels indicate whether they are carcinogenic for male and female rats.”
Graph Representation Learning via Multi-task Knowledge Distillation (2019, background): “Graph structured data are ubiquitous in application domains of machine learning, such as social networks [14, 7], chemistry [17], and biology [16, 1].”