LASSO regression
By J. Ranstam, J. Cook. British Journal of Surgery, 2018
909 citations recorded by the source corpus.
Regression models are commonly used in statistical analyses1,2. A popular use is to model the predicted risk of a likely outcome. Unfortunately, applying standard regression methods to a set of candidate variables to generate a model tends to lead to overfitting in terms of the number of variables ultimately included in the model, and also overestimation of how well the model performs in terms of using the included variables to explain the observed variability (‘optimism bias’). The model tends to perform particularly poorly with predicting observations more ‘extreme’ (very high or very low) risk. Various (penalized or regularization) regression techniques, can be used to address these problems. LASSO (Least Absolute Shrinkage and Selection Operator) regression, a shrinkage and variable selection method for regression models, is an attractive option as it addresses both problems3. Gains in computational power and incorporation into statistical software also mean that its computer-intensive nature is no longer off-putting. One area it has been used is for handing genetic data as the number of potential predictors is often large relative to the number of observations, and there is often little or no a-priori knowledge to inform variable selection. LASSO regression aims to identify the variables and corresponding regression coefficients that lead to a model that minimizes the prediction error. This is achieved by imposing a constraint on the model parameters, which ‘shrinks’ the regression coefficients towards zero, that is by forcing the sum of the absolute value of the regression coefficients to be less than a fixed value (λ). In a practical sense this constrains the complexity of the model. Variables with a regression coefficient of zero after shrinkage are excluded from the model. The choice of λ is often made by using an automated k-fold cross-validation approach. For this approach, the dataset is randomly partitioned into k sub-samples of equal size. While the k-1 sub-samples are used for developing a prediction model, the remaining sub-sample is used for validating this model. This procedure is carried out k times, with each one of the k sub-samples in turn being used for validation and the other ones for model development. An overall result is produced by combining the k separate validation results for a range of λ values and choosing the preferred λ, which is then used to determine the final model. A particular advantage with this technique is that it reduces overfitting without restricting a subset of the dataset to sole use for internal validation. LASSO regression has been shown to outperform standard methods in some settings. However, it is not a panacea to the problems of overfitting and optimism bias, and does not remove the need to validate a model in an external dataset. Additionally, the LASSO approach trades off potential bias in estimating individual parameters for a better expected overall prediction. A corresponding important disadvantage of the LASSO approach is that the regression coefficients may not be reliably interpretable in terms of independent risk factors as the focus is on the best combined prediction, not on the accuracy of the estimation and interpretation of the contribution of individual variables. Variants on the general LASSO approach exist, such as ridge regression and Elastic Net4, and the relative merits of penalization regression techniques is an area of ongoing research.
Subfields: Applied Mathematics, Control and Systems Engineering, Statistics and Probability
View Paper: https://www.semanticscholar.org/paper/76a94304f4c3eb0f173b5d22faf388260b66b31a
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Applied Mathematics
In Control and Systems Engineering
In Statistics and Probability
Citations in the corpus
- Data Augmentation in Classification and Segmentation: A Survey and New Strategies (2023, methodology): “Regularisation techniques are implemented at the model architectural level [14,15], such as dropout [16], ridge regression ( (cid:96) 2 regularisation) [17], and Lasso regression ( (cid:96) 1 regularisation) [18].”
- Unveiling differential adverse event profiles in vaccines via LLM text embeddings and ontology semantic analysis (2025, unknown): “Next, we applied LASSO regression [25] for feature selection, determining the optimal hyperparameter 𝜆 -1 𝑠𝑒 using 10-fold cross-validation.”