A Community Resource Benchmarking Predictions of Peptide Binding to MHC-I Molecules

By Bjoern Peters, H. Bui, Sune Frankild, Morten Nielsen, Claus Lundegaard, Emrah Kostem, Derek Basch, Kasper Lamberth, M. Harndahl, Ward Fleri and 5 more. PLoS Comput. Biol., 2006

308 citations recorded by the source corpus.

Recognition of peptides bound to major histocompatibility complex (MHC) class I molecules by T lymphocytes is an essential part of immune surveillance. Each MHC allele has a characteristic peptide binding preference, which can be captured in prediction algorithms, allowing for the rapid scan of entire pathogen proteomes for peptide likely to bind MHC. Here we make public a large set of 48,828 quantitative peptide-binding affinity measurements relating to 48 different mouse, human, macaque, and chimpanzee MHC class I alleles. We use this data to establish a set of benchmark predictions with one neural network method and two matrix-based prediction methods extensively utilized in our groups. In general, the neural network outperforms the matrix-based predictions mainly due to its ability to generalize even on a small amount of data. We also retrieved predictions from tools publicly available on the internet. While differences in the data used to generate these predictions hamper direct comparisons, we do conclude that tools based on combinatorial peptide libraries perform remarkably well. The transparent prediction evaluation on this dataset provides tool developers with a benchmark for comparison of newly developed prediction methods. In addition, to generate and evaluate our own prediction methods, we have established an easily extensible web-based prediction framework that allows automated side-by-side comparisons of prediction methods implemented by experts. This is an advance over the current practice of tool developers having to generate reference predictions themselves, which can lead to underestimating the performance of prediction methods they are not as familiar with as their own. The overall goal of this effort is to provide a transparent prediction evaluation allowing bioinformaticians to identify promising features of prediction methods and providing guidance to immunologists regarding the reliability of prediction tools.

Subfields: Immunology, Microbiology, Molecular Biology

View Paper: https://www.semanticscholar.org/paper/69d200bbb384f2b037e64efb28adfd489670819c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Immunology

In Microbiology

In Molecular Biology

Citations in the corpus

NetMHC-3.0: accurate web accessible predictions of human, mouse and monkey MHC class I affinities for peptides of length 8–11 (2008, methodology): “The artificial neural network (ANN) training method behind NetMHC (2,3) has been benchmarked to be the best among available methods (4).”
Distributed multi-task classification: a decentralized online learning approach (2017, methodology): “738 Mach Learn (2018) 107:727–747 Algorithm efficiency is tested also on two commonly used real-world datasets: MHC-I3 (Peters et al. 2006) contains a subset of humanMHC-I alleles (A0201,A0202,A0203,A0206, A0301, A3101, A3301, A6801, A6802) and features are extracted with bigram amino acid encoding ”
Properties of MHC Class I Presented Peptides That Enhance Immunogenicity (2013, methodology): “2 [10], the best performing predictor according to a large benchmark study [8].”
Contextual Gaussian Process Bandit Optimization (2011, methodology): “We use the data from [11], which is part of a benchmark set of MHC class I molecules [12].”
Vaxign: The First Web-Based Vaccine Design Program for Reverse Vaccinology and Applications for Vaccine Development (2010, result): “Our results indicate that Vaxign is an accurate and efficient vaccine design program.”
The Immune Epitope Database and Analysis Resource in Epitope Discovery and Synthetic Vaccine Design (2017, background): “Peptide binding datasets, far larger than any previously assembled, were originally compiled in 2006 (6).”
Gapped sequence alignment using artificial neural networks: application to the MHC class I system (2016, background): “Contact: mniel@cbs.dtu.dk Supplementary information: Supplementary data are available at Bioinformatics online.”
Large-scale validation of methods for cytotoxic T-lymphocyte epitope prediction (2007, methodology): “2 to four other publicly available methods, which have been described within the last few years: MAPPP [15], which combines proteasomal cleavage predictions with MHC class I affinity predictions, and EpiJen [16], MHCpathway [17,18], and WAPP [19], which operate with predictions of both proteasomal c”
Peptide binding predictions for HLA DR, DP and DQ molecules (2010, methodology): “This neural network based approach combines the peptide sequence representation used in the NetMHC algorithm [26,27] that was highly successful in predicting the binding specificity of HLA class I molecules [28,29] with the representation of peptide flanking residues and peptide length used in NetMH”
Asymptotically Optimal Regularization in Smooth Parametric Models (2009, methodology): “MHC-I binding prediction We evaluated our multitask regularization method on the IEDB MHC-I peptide binding dataset created by [19] and used by [13].”
A comprehensive review and performance evaluation of bioinformatics tools for HLA class I peptide-binding prediction (2020, background): “…tools; however, each study had certain limitations: either they did not include a performance evaluation of all reviewed tools, or several state-of-the-art prediction tools were not considered and benchmarked or did not have a detailed algorithm description for each of the reviewed tools [52–56].”
Novel Machine Learning Methods for MHC Class I Binding Prediction (2010, background): “[12] contains quantitative binding data (IC50 values) for various MHC alleles, including 35 human MHC alleles.”
Clustered Multi-Task Learning: A Convex Formulation (2008, methodology): “We also applied our method to the IEDB MHC-I peptide binding benchmark proposed in [10].”
Immune epitope database analysis resource (IEDB-AR) (2008, methodology): “(6), while those using independent data set are presented here for the ﬁrst time.”
NetMHCpan, a Method for Quantitative Predictions of Peptide Binding to Any HLA-A and -B Locus Protein of Known Sequence (2007, methodology): “Nonameric peptide-HLA binding data was obtained from two sources: peptide-HLA binding data recently published by Sette and coworkers [24], and data recently deposited at the IEDB by Buus and coworkers.”