Ensemble selection from libraries of models

By R. Caruana, Alexandru Niculescu-Mizil, Geoff Crew, Alex Ksikes. 2004

1,036 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

Citations in the corpus

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, background): “, 2020), including greedy selection strategies (Caruana et al., 2004; 2006; Lévesque et al., 2016; Wenzel et al., 2020).”
Hyperparameter Ensembles for Robustness and Uncertainty Quantification (2020, methodology): “This concept is already wellestablished [13] and the auto-ML community actively applies it [21, 65, 53, 46].”
Improving the Interpretability of Deep Neural Networks with Knowledge Distillation (2018, methodology): “In [31], the teacher model was built using the ensemble selection algorithm [32], which was then used to label unseen unlabeled data: the training data for the student model (also called the transfer data).”
Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine (2023, background): “Choice shuffling has an additional benefit of increasing the diversity of each reasoning path beyond temperature sampling, thereby also improving the quality of the final ensemble [5].”
Tabular Data: Deep Learning is Not All You Need (2021, methodology): “Ensemble learning is a well-known method for improving performance and reducing variance through training multiple models and combining their predictions [Caruana et al., 2004].”
Efficient and Robust Automated Machine Learning (2015, methodology): “It is well known that ensembles often outperform individual models [22, 23], and that effective ensembles can be created from a library of models [24, 25].”
Deep Boosting (2014, background): “Ensemble methods often significantly improve performance in practice (Quinlan, 1996; Bauer & Kohavi, 1999; Caruana et al., 2004; Dietterich, 2000; Schapire, 2003) and ben- Proceedings of the 31 st International Conference on Machine Learning, Beijing, China, 2014.”
Towards Automatically-Tuned Neural Networks (2016, methodology): “…goal is to make the best predictions, it is wasteful to try out dozens of machine learning models and then only use the single best model; instead, Auto-sklearn saves all models evaluated by SMAC and constructs an ensemble of these with the ensemble selection technique by Caruana et al. (2004).”
Analysis of the AutoML Challenge Series 2015-2018 (2019, methodology): “One simple and effective method to create ensembles of heterogenous models was proposed by (Caruana et al., 2004).”
Amazon SageMaker Autopilot: a white box AutoML solution at scale (2020, methodology): “, 2018) as well as ensembling (Caruana et al., 2004).”
A review of automatic selection methods for machine learning algorithms and hyper-parameter values (2016, background): “Model selection has a large body of literature and a long history in statistics and related areas, such as econometrics and quantitative social science (Claeskens and Hjort 2008; Burnham and Anderson 2003; Hendry and Doornik 2014).”
Practical Automated Machine Learning for the AutoML Challenge 2018 (2018, background): “Although ensemble building was an important component in Autosklearn, we found that ensemble selection (Caruana et al., 2004) added too many bad models to the final ensemble.”
Recent Advances in Convolutional Neural Network Acceleration (2018, background): “1"2 34%(5+($6 '#(# ,%-+.*)+ /+(01"2 78(9 :8( ;1.:#6( /+(01"2 34%(5+($6 '#(# 78(9 :8( '$-($))#($1% !"#%-<+" Figure 4: Illustration of knowledge distillation. [73]. The way that knowledge is transferred can be depicted in the following Figure4. Information ﬂo”