ℓ1-penalization for mixture regression models

By Nicolas Städler, Peter Buhlmann, Sara A. van de Geer. 2010

323 citations recorded by the source corpus.

We consider a finite mixture of regressions (FMR) model for high-dimensional inhomogeneous data where the number of covariates may be much larger than sample size. We propose an ℓ1-penalized maximum likelihood estimator in an appropriate parameterization. This kind of estimation belongs to a class of problems where optimization and theory for non-convex functions is needed. This distinguishes itself very clearly from high-dimensional estimation with convex loss- or objective functions as, for example, with the Lasso in linear or generalized linear models. Mixture models represent a prime and important example where non-convexity arises.For FMR models, we develop an efficient EM algorithm for numerical optimization with provable convergence properties. Our penalized estimator is numerically better posed (e.g., boundedness of the criterion function) than unpenalized maximum likelihood estimation, and it allows for effective statistical regularization including variable selection. We also present some asymptotic theory and oracle inequalities: due to non-convexity of the negative log-likelihood function, different mathematical arguments are needed than for problems with convex losses. Finally, we apply the new method to both simulated and real data.

Subfields: Artificial Intelligence, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/19f90a8c59b2b5924debf81fc38c53f2f445b236

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Prediction error of cross-validated Lasso (2015, methodology): “ literature on signiﬁcance tests for Lasso [29, 32]. Signiﬁcance tests almost always require some plug-in estimate of σ2. There are a number of proposed methods for error variance estimation in Lasso [15, 20, 38, 41, 42, 43]. The recent paper of Reid, Tibshirani and Friedman [36] gives a comprehensi”
SLOPE—ADAPTIVE VARIABLE SELECTION VIA CONVEX OPTIMIZATION (2014, methodology): “When p ≥ n, some solutions for simultaneous estimation of σ and regression coefficients using `1 optimization schemes were proposed, see e.g., Städler et al. (2010) and Sun and Zhang (2012).”
Confidence intervals for low dimensional parameters in high dimensional linear models (2011, background): “An alternative, larger estimate of σ, producing more conservative approximate confidence intervals, is the penalized maximum likelihood estimator of Städler et al. (2010).”
Challenges of Big Data Analysis (2013, unknown): “In high dimensions, however, we need to carefully regularize the estimating procedure to avoid overfitting or noise accumulation and to devise good computation algorithms [70,71].”
Estimation for High‐Dimensional Linear Mixed‐Effects Models Using ℓ1‐Penalization (2010, methodology): “(B.1) By the arguments in Städler et al. (2010), for a constant c0 independent of N , n, p and the design, Ē(φ̂|φ0) ≥ (β̂ − β0)TΣN,n(β̂ − β0)/c20 + ‖η̂ − η0‖22/c20.”
High-dimensional regression with unknown variance (2011, methodology): “There has been some recent work [16, 68, 71] to modify procedures like the Lasso in such a way that the tuning parameter does not depend anymore on σ2 (see Section 4.”
Scaled sparse linear regression (2011, methodology): “St¨adler et al. (2010) proposed to estimate the regression coefﬁcients and noise level by maximizing their joint log-likelihood with an (cid:96) 1 penalty on the regression coefﬁcients.”