Sequential Model-Based Ensemble Optimization

By Alexandre Lacoste, Hugo Larochelle, Mario Marchand, François Laviolette. 2014

37 citations recorded by the source corpus.

One of the most tedious tasks in the application of machine learning is model selection, i.e. hyperparameter selection. Fortunately, recent progress has been made in the automation of this process, through the use of sequential model-based optimization (SMBO) methods. This can be used to optimize a cross-validation performance of a learning algorithm over the value of its hyperparameters. However, it is well known that ensembles of learned models almost consistently outperform a single model, even if properly selected. In this paper, we thus propose an extension of SMBO methods that automatically constructs such ensembles. This method builds on a recently proposed ensemble construction paradigm known as agnostic Bayesian learning. In experiments on 22 regression and 39 classification data sets, we confirm the success of this proposed approach, which is able to outperform model selection with SMBO.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Optimization & Training

Citations in the corpus

Hyperparameter optimization with approximate gradient (2016, methodology): “This probabilistic model typically relies on a Gaussian process regressor but other approaches exist using trees (Bergstra et al., 2011) or ensemble methods (Lacoste et al., 2014).”
Benchmark and Survey of Automated Machine Learning Frameworks (2019, neutral): “Instead of only yielding the best performing configuration, the set of best performing configurations can be used to create an ensemble (Lacoste et al., 2014; Feurer et al., 2015a; Wistuba et al., 2017).”
Survey on Automated Machine Learning (2019, positive): “Instead of only yielding the best performing configuration, the set of best performing configurations can be used to create an ensemble (Lacoste et al., 2014; Feurer et al., 2015a; Wistuba et al., 2017).”
Efficient Progressive Neural Architecture Search (2018, methodology): “In the context of machine learning, this method has been used to automatically con-2 struct ensembles of classiﬁers by Lacoste et al ., [13].”
A review of automatic selection methods for machine learning algorithms and hyper-parameter values (2016, methodology): “In fact, some automatic selection methods were designed specifically to address the issue of automatically forming and selecting ensembles (Thornton et al. 2013; Lacoste et al. 2014a, b).”
Automatic Frankensteining: Creating Complex Ensembles Autonomously (2017, background): “While some research already identified that an average ensemble improves the performance [11, 5, 12], no one has looked into the direction of deep ensembles that consist of several layers.”
Hyperparameters Adaptation for Restricted Boltzmann Machines Based on Free Energy (2016, unknown): “But random search is still computational expensive.”