Bayesian Hyperparameter Optimization for Ensemble Learning

By Julien-Charles Levesque, Christian Gagné, Robert Sabourin. 2016

63 citations recorded by the source corpus.

In this paper, we bridge the gap between hyperparameter optimization and ensemble learning by performing Bayesian optimization of an ensemble with regards to its hyperparameters. Our method consists in building a fixed-size ensemble, optimizing the configuration of one classifier of the ensemble at each iteration of the hyperparameter optimization algorithm, taking into consideration the interaction with the other models when evaluating potential performances. We also consider the case where the ensemble is to be reconstructed at the end of the hyperparameter optimization phase, through a greedy selection over the pool of models generated during the optimization. We study the performance of our proposed method on three different hyperparameter spaces, showing that our approach is better than both the best single model and a greedy ensemble construction over the models produced by a standard Bayesian optimization.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Probabilistic & Bayesian Methods, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Frank Hutter
2. Quoc V. Le
3. Ali Farhadi
4. Martin Wistuba
5. Percy Liang
6. Stefan Falkner
7. Ludwig Schmidt
8. Aaron Klein

In Computational Theory and Mathematics

1. Frank Hutter
2. Stefan Falkner
3. Aaron Klein
4. Martin Wistuba
5. Lars Schmidt-Thieme
6. Nicolas Schilling
7. Marco F. Huber
8. Marc-André Zöller

In Probabilistic & Bayesian Methods

1. Frank Hutter
2. Aaron Klein
3. Stefan Falkner
4. Martin Wistuba
5. Rodolphe Jenatton
6. Lars Schmidt-Thieme
7. Ali Farhadi
8. Jasper Snoek

In Neural Architecture Search & AutoML

1. Frank Hutter
2. Aaron Klein
3. Stefan Falkner
4. Martin Wistuba
5. Lars Schmidt-Thieme
6. Rodolphe Jenatton
7. Nicolas Schilling
8. Jasper Snoek

Citations in the corpus

• Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, background): “, 2020), including greedy selection strategies (Caruana et al., 2004; 2006; Lévesque et al., 2016; Wenzel et al., 2020).”
• BOHB: Robust and Efficient Hyperparameter Optimization at Scale (2018, background): “, 2016), active ensembling to combine models found during the optimization (Lévesque et al., 2016), and using multiple fidelities (Swersky et al.”
• Automatic Frankensteining: Creating Complex Ensembles Autonomously (2017, methodology): “Finally, Levesque [12] proposes to use Bayesian optimization directly to estimate which prediction model is the best candidate to be added to the ensemble.”
• Hyperparameter Ensembles for Robustness and Uncertainty Quantification (2020, background): “Finally, recent research in Bayesian optimization has also focused on directly optimizing the performance of the ensemble while tuning the hyperparameters [46].”