Getting the Most Out of Ensemble Selection

By R. Caruana, Art Munson, Alexandru Niculescu-Mizil. 2006

170 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Information Systems, Optimization & Training, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Optimization & Training

In Evaluation & Benchmarks

Citations in the corpus

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, methodology): “Finally, previous work has explored building ensembles of models produced by hyperparameter searches [83, 64, 78], including greedy selection strategies [15, 16, 56, 93].”
Automated Machine Learning: From Principles to Practices (2018, methodology): “Concretely, In the the two-stage paradigm method [10], the one-shot NAS process involves training the complete supernet in the first stage, and in the second stage, it evaluates specific architectures by zeroing out large portions of the one-shot model with a designed dropout strategy.”
Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning (2020, methodology): “The AutoML system then returns a single model in case of holdout, and a combination of K models in case of K-fold cross-validation (Caruana et al., 2006).”
Practical Automated Machine Learning for the AutoML Challenge 2018 (2018, methodology): “Therefore, we introduced a new regularization technique similar to library pruning (Caruana et al., 2006).”
Efficient and Robust Automated Machine Learning (2015, unknown): “It is well known that ensembles often outperform individual models [22, 23], and that effective ensembles can be created from a library of models [24, 25].”
Accurate predictions on small data with a tabular foundation model (2025, unknown): “We learn the weights using greedy ensemble selection (GES) 42,66 with 25 iterations on prediction data from holdout validation.”
Benchmarking state-of-the-art classification algorithms for credit scoring: An update of research (2015, background): “The clear winner among the heterogeneous ensembles is HCES-Bag (Caruana et al., 2006).”
Efﬁcient and Robust Automated Machine Learning (2015, unknown): “It is well known that ensembles often outperform individual models [22, 23], and that effective ensembles can be created from a library of models [24, 25].”
Hyperparameter Ensembles for Robustness and Uncertainty Quantification (2020, background): “In particular, the greedy ensemble construction from [13] (and later variations thereof [12]) was shown to work best among other algorithms, either more expensive or more prone to overfitting.”
Auto-sklearn: Efficient and Robust Automated Machine Learning (2019, background): “1 It is well known that ensembles often outperform individual models [18, 24], and that eﬀective ensembles can be created from a library of models [9, 8].”
Multimodal fusion for multimedia analysis: a survey (2010, background): “The fusion problems have also been addressed in other domains such as machine learning [48], data mining [24] and information retrieval [133], however, the focus of this paper is restricted to the multimedia research domain.”
Data-driven exemplar model selection (2014, background): “[2, 3] show that such a selection strategy is prone to overfitting.”
Regularizing Deep Convolutional Neural Networks with a Structured Decorrelation Constraint (2016, methodology): “To reduce overfitting, several regularizers, optimization methods and network structures have been proposed, including Early Stopping [13], Weight Decay [14], Model Ensemble [15–17], Data Augmentation [18], Dropout [19, 20], DropConnect [21], Maxout [22], Batch Normalization [23], Residual Networks…”