Identifying a Minimal Class of Models for High-dimensional Data

By Daniel Nevo, Yaacov Ritov. 2015

19 citations recorded by the source corpus.

Model selection consistency in the high-dimensional regression setting can be achieved only if strong assumptions are fulfilled. We therefore suggest to pursue a different goal, which we call a minimal class of models. The minimal class of models includes models that are similar in their prediction accuracy but not necessarily in their elements. We suggest a random search algorithm to reveal candidate models. The algorithm implements simulated annealing while using a score for each predictor that we suggest to derive using a combination of the Lasso and the Elastic Net. The utility of using a minimal class of models is demonstrated in the analysis of two datasets.

Subfields: Artificial Intelligence, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Forward stability and model path selection (2021, neutral): “Nevo and Ritov (2017) also explored the high-dimensional linear regression setting and proposed a random search procedure utilizing simulated annealing.”
All Models are Wrong, but Many are Useful: Learning a Variable’s Importance by Studying an Entire Class of Prediction Models Simultaneously (2018, background): “Nevo and Ritov (2017) report interpretations corresponding to a set of well-fitting, sparse linear models.”
A study in Rashomon curves and volumes: A new perspective on generalization and model simplicity in machine learning (2019, methodology): “Rashomon sets: Rashomon sets have been used for various purposes (Breiman et al., 2001; Srebro et al., 2010; Fisher et al., 2019; Coker et al., 2018; Tulabandhula and Rudin, 2014b; Meinshausen and Bühlmann, 2010; Letham et al., 2016; Nevo and Ritov, 2017).”
All Models are Wrong but many are Useful: Variable Importance for Black-Box, Proprietary, or Misspecified Prediction Models, using Model Class Reliance (2018, background): “Nevo and Ritov (2015) report interpretations corresponding to a set of well-fitting, sparse linear models.”
Variable Importance Clouds: A Way to Explore Variable Importance for the Set of Good Models (2019, background): “nsistent with the historical data. The elds of robust optimization and chance-constrained programming use uncertainty sets to make decisions; these uncertainty sets could arise from the Rashomon set. Nevo and Ritov (2017) analyze a model selection problem in high-dimensional regression and identify ”