Variable Selection using MM Algorithms.

By David R. Hunter, Runze Li. Annals of Statistics, 2005

498 citations recorded by the source corpus.

Variable selection is fundamental to high-dimensional statistical modeling. Many variable selection techniques may be implemented by maximum penalized likelihood using various penalty functions. Optimizing the penalized likelihood function is often challenging because it may be nondifferentiable and/or nonconcave. This article proposes a new class of algorithms for finding a maximizer of the penalized likelihood for a broad class of penalty functions. These algorithms operate by perturbing the penalty function slightly to render it differentiable, then optimizing this differentiable function using a minorize-maximize (MM) algorithm. MM algorithms are useful extensions of the well-known class of EM algorithms, a fact that allows us to analyze the local and global convergence of the proposed algorithm using some of the techniques employed for EM algorithms. In particular, we prove that when our MM algorithms converge, they must converge to a desirable point; we also discuss conditions under which this convergence may be guaranteed. We exploit the Newton-Raphson-like aspect of these algorithms to propose a sandwich estimator for the standard errors of the estimators. Our method performs well in numerical tests.

Subfields: Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/0bca2c089baa646d15823343b379c32e34f6fdf7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

Citations in the corpus

Shrinkage Estimation of the Varying Coefficient Model (2009, background): “Then, the loss function in (4) can be locally approximated by (Fan and Li 2001; Hunter and Li 2005)Xn t¼1 Xn i¼1 Yi X>i bðZtÞ 2 KhðZt ZiÞ þ Xd j¼1 lj k bj k2 k b̂ðmÞl; j k ¼ Xn t¼1 Xn i¼1 Yi X>i bðZtÞ 2 KhðZt ZiÞ þ Xd j¼1 lj b2j ðZtÞ k b̂ðmÞl; j k 0@ 1A; whose minimizer is given by B̂ ðmþ1Þ l with…”
ADAPTIVE SEMI-VARYING COEFFICIENT MODEL SELECTION (2012, unknown): “Then, the loss function in (2.1) can be locally approximated by (Fan and Li (2001); Hunter and Li (2005)) n∑ t=1 n∑ i=1 { Yi −X⊤i β(Zt) }2 Kh(Zt − Zi) + d∑ j=1 λj ∥bj∥2 ∥b̂(m)λ,j ∥ .”
PENALIZED LINEAR UNBIASED SELECTION (2007, methodology): “Iterative algorithms were developed there and in Hunter and Li (2005) and Zou and Li (2006) to approximate a local minimizer of the SCAD penalized loss for fixed (λ, γ).”
NONCONCAVE PENALIZED M-ESTIMATION WITH A DIVERGING NUMBER OF PARAMETERS (2011, methodology): “To avoid this drawback, Hunter and Li (2005) suggested optimizing a slightly perturbed version of (4.”
STRONG ORACLE OPTIMALITY OF FOLDED CONCAVE PENALIZED ESTIMATION (2012, background): “Both LQA and LLA are related to the MM principle (Hunter and Lange, 2004; Hunter and Li, 2005).”
Nearly unbiased variable selection under minimax concave penalty (2010, methodology): “Iterative algorithms were developed there and in Hunter and Li (2005) and Zou and Li (2008) to approximate a local minimizer of the SCAD penalized loss for fixed (λ,γ).”
Variable Selection in Finite Mixture of Regression Models (2007, methodology): “For other issues of numerical implementation, the work of Hunter and Li (2005) will be helpful.”
Challenges of Big Data Analysis (2013, unknown): “Thiscreates issuesofheterogeneity,experimentalvariationsand statisticalbiases,andrequiresustodevelopmore adaptiveandrobustprocedures.”
VARIABLE SELECTION IN QUANTILE REGRESSION (2009, methodology): “For the MM algorithm, we set Hunter and Li (2005)’s parameter τ to be 10−6 in their Equation (3.12).”
Asymptotic properties of bridge estimators in sparse high-dimensional regression models (2008, background): “Examples include Van der Laan and Bryan (2001), Bühlmann (2006), Fan, Peng and Huang (2005), Huang, Wang and Zhang (2005), Huang and Zhang (2005) and Kosorok and Ma (2007). Fan and Li (2006) provide a review of statistical challenges in high-dimensional problems that arise in many important applicat”
A General Theory of Concave Regularization for High-Dimensional Sparse Estimation Problems (2011, methodology): “In summary, our results imply the following: under appropriate 2 regularity conditions, plus appropriate assumptions on the penalty ρ(t;λ), procedures considered earlier such as MCP [42] or multi-stage convex relaxation [19, 46, 51] give the same local solution that is also the global minimizer of (”
SCAD-penalized regression in high-dimensional partially linear models (2009, methodology): “Hunter and Li (2005) suggested using Simulation in the next section shows that this algorithm works very well for our problem.”
Regularization Parameter Selections via Generalized Information Criterion (2010, methodology): “…studied in the recent literature: (i) Fan and Li (2001) proposed the local quadratic approximation (LQA) al-gorithm, which was further analyzed by Hunter and Li (2005); (ii) Efron et al. (2004) introduced the LARS algorithm, which can be used for the adaptive LASSO (Zou 2006; Wang, Li, and Tsai…”
The Adaptive Lasso and Its Oracle Properties (2006, methodology): “Our results offer new insights into the (cid:1) 1 -related methods and the use of the (cid:1) 1 penalty in statistical modeling.”
A unified approach to model selection and sparse recovery using regularized least squares (2009, background): “Hunter and Li (2005) propose and study MM algorithms for variable selection.”
Regularized M-estimators with nonconvexity: statistical and algorithmic theory for local optima (2013, methodology): “Various methods have been proposed for optimizing convex loss functions with nonconvex penalties [3, 4, 15], but these methods are only guaranteed to generate local optima of the composite objective, which have not been proven to be well-behaved.”
On the Convergence of the Concave-Convex Procedure (2009, background): “The MM principle appears in many places in statistical computation, including multidimensional scaling [6], robust regression [14], correspondence analysis [13], variable selection [16], sparse signal recovery [4], etc.”