Penalized classification using Fisher’s linear discriminant

By Daniela M. Witten, Robert Tibshirani. Journal of The Royal Statistical Society Series B-statistical Methodology, 2011

433 citations recorded by the source corpus.

Summary. We consider the supervised classification setting, in which the data consist of p features measured on n observations, each of which belongs to one of K classes. Linear discriminant analysis (LDA) is a classical method for this problem. However, in the high dimensional setting where p≫n, LDA is not appropriate for two reasons. First, the standard estimate for the within‐class covariance matrix is singular, and so the usual discriminant rule cannot be applied. Second, when p is large, it is difficult to interpret the classification rule that is obtained from LDA, since it involves all p features. We propose penalized LDA, which is a general approach for penalizing the discriminant vectors in Fisher's discriminant problem in a way that leads to greater interpretability. The discriminant problem is not convex, so we use a minorization–maximization approach to optimize it efficiently when convex penalties are applied to the discriminant vectors. In particular, we consider the use of L1 and fused lasso penalties. Our proposal is equivalent to recasting Fisher's discriminant problem as a biconvex problem. We evaluate the performances of the resulting methods on a simulation study, and on three gene expression data sets. We also survey past methods for extending LDA to the high dimensional setting and explore their relationships with our proposal.

Subfields: Computer Vision and Pattern Recognition, Molecular Biology, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/7f98b5ad97c6cdb89d17ef94c72bc65abaacc188

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Molecular Biology

In Statistics and Probability

Citations in the corpus

A Unified Algorithmic Framework for Block-Structured Optimization Involving Big Data: With applications in machine learning and signal processing (2015, methodology): “As suggested in [82], one can linearize the first part of the objective in (26) iteratively to obtain a tight upper bound of the objective.”
A Simple Two-Sample Test in High Dimensions Based on L2-Norm (2020, background): “And, these ideas have been extended to other problems such as classification and discriminant analysis (Cui, Li, and Zhong 2015; Fan and Fan 2008; Fan, Feng, and Tong 2012; Fan et al. 2015; Witten and Tibshirani 2011).”
Applied Predictive Modeling (2013, methodology): “The lasso model has been extended to many other techniques, such as linear discriminant analysis (Clemmensen et al. 2011; Witten and Tibshirani 2011), PLS (Chun and Keleş 2010), and PCA (Jolliffe et al.”
A tutorial review: Metabolomics and partial least squares-discriminant analysis--a marriage of convenience or a shotgun wedding. (2015, neutral): “[27,98–106].”
Random matrix theory in statistics: A review (2014, methodology): “…is imposed either through sparse penalization or thresholding of the discriminant function, often in combination with sparse estimation of the precision matrix, have been proposed and analyzed by Cai and Liu (2011b), Clemmensen et al. (2011), Shao et al. (2011), Witten and Tibshirani (2009, 2011).”