SLOPE—ADAPTIVE VARIABLE SELECTION VIA CONVEX OPTIMIZATION

By Malgorzata Bogdan, Ewout van den Berg, Chiara Sabatti, Weijie Su, E.J. Candes. 2014

318 citations recorded by the source corpus.

We introduce a new estimator for the vector of coefficients β in the linear model y = Xβ + z, where X has dimensions n × p with p possibly larger than n. SLOPE, short for Sorted L-One Penalized Estimation, is the solution to minb∈ℝp12‖y−Xb‖ℓ22+λ1|b|(1)+λ2|b|(2)+⋯+λp|b|(p),where λ1 ≥ λ2 ≥ … ≥ λp ≥ 0 and |b|(1)≥|b|(2)≥⋯≥|b|(p) are the decreasing absolute values of the entries of b. This is a convex program and we demonstrate a solution algorithm whose computational complexity is roughly comparable to that of classical ℓ1 procedures such as the Lasso. Here, the regularizer is a sorted ℓ1 norm, which penalizes the regression coefficients according to their rank: the higher the rank—that is, stronger the signal—the larger the penalty. This is similar to the Benjamini and Hochberg [J. Roy. Statist. Soc. Ser. B 57 (1995) 289–300] procedure (BH) which compares more significant p-values with more stringent thresholds. One notable choice of the sequence {λi} is given by the BH critical values λBH(i)=z(1−i⋅q/2p), where q ∈ (0, 1) and z(α) is the quantile of a standard normal distribution. SLOPE aims to provide finite sample guarantees on the selected model; of special interest is the false discovery rate (FDR), defined as the expected proportion of irrelevant regressors among all selected predictors. Under orthogonal designs, SLOPE with λBH provably controls FDR at level q. Moreover, it also appears to have appreciable inferential properties under more general designs X while having substantial power, as demonstrated in a series of experiments running on both simulated and real data.

Subfields: Computational Theory and Mathematics, Control and Systems Engineering, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Theory and Mathematics

In Control and Systems Engineering

In Statistics and Probability

Citations in the corpus

Approximate residual balancing: debiased inference of average treatment effects in high dimensions (2016, methodology): “Through our analysis, we assume that β̂c is obtained via the lasso; however, we could just as well consider, for example, the square-root lasso (Belloni et al., 2011), sorted L1-penalized regression (Bogdan et al., 2015; Su and Candes, 2016) or other methods with comparable L1-risk bounds.”
On Regularizing Rademacher Observation Losses (2016, methodology): “Modulo some assumptions, SLOPE yields a control the false discovery rate (FDR) — i.e., negligible coefficients in the "true” linear model θ∗ that are found significant in the learned θ —.”
Adaptive Huber Regression (2017, background): “As pointed out by a reviewer, if one pursues a sparsity-adaptive approach, such as the SLOPE (Bogdan et al. 2015; Bellec, Lecué, and Tsybakov 2018), the upper bound on (cid:5) 2 -error can be improved from (cid:25) s log ( d )/ n to (cid:25) s log ( ed / s )/ n .”
Sequential selection procedures and false discovery rate control (2013, background): “This question is discussed in more detail in several papers (e.g., Benjamini and Gavrilov, 2009; Bogdan et al., 2014; G’Sell et al., 2013; Lin et al., 2011; Wu et al., 2007).”
Sparse Estimation with Strongly Correlated Variables using Ordered Weighted L1 Regularization (2014, result): “Our results extend previous ones for OSCAR in several ways: for the squared error loss, our conditions hold for the more general OWL norm and under weaker assumptions; we also establish clutering conditions for the absolute error loss, which is, as far as we know, a novel result.”
Robust machine learning by median-of-means: Theory and practice (2017, result): “le inequalities, in [64,99,6] for support recovery results; more results and references on LASSO can be found in the books [19,44]. B.2 SLOPE SLOPE is an extension of the LASSO that was introduced in [16,84]. The class Fis still F= f t; : t2 Rdgand the regularization function is dened for parameters”
Slope meets Lasso: Improved oracle bounds and optimality (2016, background): “The Slope estimator suggested in [6] is defined as a solution of the convex minimization problem given in (2.”
Regularization, sparse recovery, and median-of-means tournaments (2017, result): “rnament procedure and the technical machinery we require, let us describe in more detail the two applications mentioned previously, both of which originating in sparse recovery: lasso, and slope(see, [1, 3, 14, 15]). The two are regularized empirical risk minimization procedures in Rd, and we would ”
SLOPE is Adaptive to Unknown Sparsity and Asymptotically Minimax (2015, methodology): “Under independence, i.e. y ∼ N ( β , σ 2 I p ), SLOPE controls the FDR [15], namely, FDR(SLOPE) ≤ qp 0 /p , where again p 0 is the number of nulls, i.e. of vanishing means.”
Learning Games and Rademacher Observations Losses (2015, methodology): “= θ⊤Γθ Ridge ‖θ‖∞ .= maxk |θk| ℓ∞ ‖θ‖Φ .= maxM∈Sd(M|θ|)⊤ξ SLOPE (33) (Bach et al., 2011; Bogdan et al., 2015; Duchi & Singer, 2009;Su & Candès, 2015).”
Panning for Gold: Model-free Knockoffs for High-dimensional Controlled Variable Selection (2016, methodology): “For binary covariates, the causal inference 1One notable exception is the SLOPE procedure (Bogdan et al., 2015), which acts like a regression analogue of BHq without ever computing p-values, and provably controls the FDR in homoscedastic linear regression when the design matrix has orthogonal column”
A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights (2014, methodology): “This is a recently introduced testing and estimation procedure (Bogdan et al., 2015).”
Familywise error rate control via knockoffs (2015, background): “, 2004) and sorted-`1-penalized estimation (Bogdan et al., 2015; Su and Candès, 2015), can be used instead as long as they obey the sufficiency and antisymmetry properties defined therein.”
A Critical Review of LASSO and Its Derivatives for Variable Selection Under Dependence Among Covariates (2020, methodology): “Furthermore, more recent techniques like the Knockoﬀ ﬁlter (Barber et al. (2015), Candes et al. (2018)) or SLOPE (Bogdan et al. (2015)) have been introduced to control some measures of the type I error.”