Collaborative hyperparameter tuning

By Rémi Bardenet, Mátyás Brendel, Balázs Kégl, Michèle Sebag. 2013

384 citations recorded by the source corpus.

Subfields: Optimization & Training, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

In Neural Architecture Search & AutoML

Citations in the corpus

Automated Machine Learning: State-of-The-Art and Open Challenges (2019, methodology): “When computing the similarity between two tasks represented as two feature vectors of meta data, it is important to normalize these vectors or apply dimensionality reduction techniques such as principle component analysis [15, 5].”
Transfer Learning with Neural AutoML (2018, neutral): “There is also prior research on transfer of optimizers for Neural AutoML; Sequential Model-based Optimizers have been transferred across tasks to improve hyperparameter tuning [31, 32], we propose a parallel solution for neural methods.”
Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets (2016, background): “For example, in several studies, it found better instantiations of convolutional network hyperparameters than domain experts, repeatedly improving the top score on the CIFAR-10 [12] benchmark without data augmentation [5, 13, 11].”
Efficient Benchmarking of Hyperparameter Optimizers via Surrogates (2015, methodology): “In fact, several researchers have already applied this approach to simplify experiments (Bardenet et al. 2013; Snoek, Larochelle, and Adams 2012; Birattari et al. 2002).”
Taking the Human Out of the Loop: A Review of Bayesian Optimization (2016, background): “property within the Bayesian optimization framework [12], [49], [79], [89], [148], [160].”
Meta-Learning: A Survey (2018, neutral): “SCoT [9] trains a single surrogate ranking model f : M × → R, predicting the rank of θi on task tj .”
Initializing Bayesian Hyperparameter Optimization via Meta-Learning (2015, methodology): “…can be divided into at least five groups: • Simple metafeatures, such as the number of features, patterns or classes, describe the basic dataset structure (Michie et al. 1994; Kalousis 2002) • PCA metafeatures (Bardenet et al. 2013) compute various statistics of the datasets principal components.”
Input Warping for Bayesian Optimization of Non-Stationary Functions (2014, methodology): “Other approaches include Bardenet et al. [2013], which ﬁnds a joint latent function over tasks explicitly using a ranking model, and Hutter et al. [2011] which uses a set of auxiliary task features to improve prediction.”
Transferable AutoML by Model Sharing Over Grouped Datasets (2019, methodology): “Unlike multi-task solutions [35, 8, 4] designed for Bayesian optimization, or transfer learning with AutoML [37] based on reinforcement learning, our approach can be easily combined with most existing AutoML techniques in an out-of-box fashion.”
Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms (2012, neutral): “Categories and Subject Descriptors I.2.6 [Artificial Intelligence]: Learning; I.2.2 [Artificial Intelligence]: Automatic Programming; G.1.6 [Mathematics of Computing]: Optimization General Terms Algorithms, Performance, Experimentation Keywords Model selection; Hyperparameter optimization; WEKA”
Auto-WEKA : combined selection and hyperparameter optimization of supervised machine learning algorithms (2014, background): “Meta-learning is a discipline that uses machine learning to make predictions about a dataset as a whole, rather than a particular element in the dataset [Bardenet et al., 2013, Leite et al., 2012, Pfahringer et al., 2000, Vilalta and Drissi, 2002].”
Analysis of the AutoML Challenge Series 2015-2018 (2019, methodology): “Recently, (Bardenet et al., 2013) proposed to apply collaborative filtering methods to model search.”
Fast Bayesian hyperparameter optimization on large datasets (2017, methodology): “…has driven interest in more sophisticated hyperparameter optimization methods over the past years (Bergstra et al., 2011; Hutter et al., 2011; Bergstra and Bengio, 2012; Snoek et al., 2012; Bardenet et al., 2014; Bergstra et al., 2014; Swersky et al., 2013, 2014; Snoek et al., 2014, 2015).”
Speeding up the Hyperparameter Optimization of Deep Convolutional Neural Networks (2018, methodology): “As a result, sequential model-based optimization (SMBO) algorithms have been employed in many settings when the performance evaluation of a model is expensive (Hutter et al., 2009, 2011; Bergstra et al., 2011; Brendel and Schoenauer, 2011; Bardenet et al., 2013; Thornton et al., 2013).”
Meta-QSAR: a large-scale application of meta-learning to drug design and discovery (2017, methodology): “Finally, algorithms can also be ranked using collaborative filtering (Bardenet et al. 2013; Misir and Sebag 2013; Smith et al. 2014b).”
Fair lending needs explainable models for responsible recommendation (2018, background): “We posit that landmark features and principal components, while popular as meta-features for meta-learning [5, 26], are less intuitive and hence more difficult to use in satisfactory explanations. locality in the space of directly measured features is an important factor that explains…”
Insights on Transfer Optimization: Because Experience is the Best Teacher (2018, background): “Along the same lines as the above, in [42] the common response surface was constructed by considering, for a given”
Bayesian Optimization with Tree-structured Dependencies (2017, neutral): “We say that the rank of a method is equal to i if it performs the ith best (see, e.g., Bardenet et al. (2013); Feurer et al. (2015)).”