Double-check: validation of diagnostic statistics for PLS-DA models in metabolomics studies

By E. Szymańska, Edoardo Saccenti, Age K. Smilde, Johan A. Westerhuis. Metabolomics, 2011

806 citations recorded by the source corpus.

Partial Least Squares-Discriminant Analysis (PLS-DA) is a PLS regression method with a special binary ‘dummy’ y-variable and it is commonly used for classification purposes and biomarker selection in metabolomics studies. Several statistical approaches are currently in use to validate outcomes of PLS-DA analyses e.g. double cross validation procedures or permutation testing. However, there is a great inconsistency in the optimization and the assessment of performance of PLS-DA models due to many different diagnostic statistics currently employed in metabolomics data analyses. In this paper, properties of four diagnostic statistics of PLS-DA, namely the number of misclassifications (NMC), the Area Under the Receiver Operating Characteristic (AUROC), Q2 and Discriminant Q2 (DQ2) are discussed. All four diagnostic statistics are used in the optimization and the performance assessment of PLS-DA models of three different-size metabolomics data sets obtained with two different types of analytical platforms and with different levels of known differences between two groups: control and case groups. Statistical significance of obtained PLS-DA models was evaluated with permutation testing. PLS-DA models obtained with NMC and AUROC are more powerful in detecting very small differences between groups than models obtained with Q2 and Discriminant Q2 (DQ2). Reproducibility of obtained PLS-DA models outcomes, models complexity and permutation test distributions are also investigated to explain this phenomenon. DQ2 and Q2 (in contrary to NMC and AUROC) prefer PLS-DA models with lower complexity and require higher number of permutation tests and submodels to accurately estimate statistical significance of the model performance. NMC and AUROC seem more efficient and more reliable diagnostic statistics and should be recommended in two group discrimination metabolomic studies.

Subfields: Analytical Chemistry, Molecular Biology

View Paper: https://www.semanticscholar.org/paper/b20cefd0c81f44365dd457df934164521daec0cc

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Analytical Chemistry

In Molecular Biology

Citations in the corpus

A fully integrated computer-aided diagnosis system for digital X-ray mammograms via deep learning detection, segmentation, and classification (2018, methodology): “Then, the final performance for all of these models is only assessed utilizing the testing set [19,48,61].”
Metabolomics and Multi-Omics Integration: A Survey of Computational Methods and Resources (2020, background): “These metrics are compared for statistical significance in [153].”
Simultaneous Detection of Multiple Adulterants in Ground Roasted Coffee by ATR-FTIR Spectroscopy and Data Fusion (2017, background): “…corresponds to the ratio between true positive samples (TP) and the sum of false negative samples (FN) and the total number of known positive samples (TP) (Szymańska et al. 2012): SE ¼ TP TPþ FNð Þ ð2Þ Other important figures of merit are the number of misclassifications (NMC) and the reliability…”
metaX: a flexible and comprehensive software for processing metabolomics data (2017, methodology): “…partial least squares-discriminant analysis (OPLS-DA), with numerical and graphical results and diagnostics (optimal number of components estimated by cross-validation, R 2 , Q 2 , variable importance in projection (VIP), statistical significance of the model by permutation testing) [22].”
A tutorial review: Metabolomics and partial least squares-discriminant analysis--a marriage of convenience or a shotgun wedding. (2015, methodology): “This approach has also been extensively used in “omics” related fields, for example: metabolomics [3], proteomics [4], genomics [5,6] and in many other fields which generates large amounts of data such as spectroscopy, as described elsewhere [7].”
Analytical Methods in Untargeted Metabolomics: State of the Art in 2015 (2015, positive): “…require optimization (i.e., optimal number of PLS/O- PLS components to be used) a double cross-validation schema is usually required: a first cross-validation step is applied to optimize the model and a second step for assessing the model quality (Westerhuis et al., 2008; Szymanska et al., 2012).”
Development of Artificial Intelligence for Determining Major Depressive Disorder Based on Resting-State EEG and Single-Pulse Transcranial Magnetic Stimulation-Evoked EEG Indices (2024, methodology): “120 cases in the present study, we used five-fold outer loop double cross-validation [29,30] to evaluate the estimation performance of We applied machine learning to these feature sets to create a model that can discriminate whether the participant has depression or not based on EEG data. the model.”