Efficient Transfer Learning Method for Automatic Hyperparameter Tuning

By Dani Yogatama, Gideon S. Mann. 2014

226 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, background): “Furthermore, transfer learning is supported by Google Vizier to accelerate the study of warm starting the model training with useful information from relevant studies (Yogatama and Mann, 2014).”
Transfer Learning with Neural AutoML (2018, neutral): “There is also prior research on transfer of optimizers for Neural AutoML; Sequential Model-based Optimizers have been transferred across tasks to improve hyperparameter tuning [31, 32], we propose a parallel solution for neural methods.”
Taking the Human Out of the Loop: A Review of Bayesian Optimization (2016, background): “There have been several attempts to exploit this property within the Bayesian optimization framework [89], [79], [12], [148], [161], [49].”
Google Vizier: A Service for Black-Box Optimization (2017, background): “Yogatama and Mann [32] propose a more efficient approach, which scales as Θ(kn + n3) for k studies of n trials each, where the cubic term comes from using a Gaussian process in their acquisition function.”
Initializing Bayesian Hyperparameter Optimization via Meta-Learning (2015, methodology): “These collaborative SMBO methods (Bardenet et al. 2013; Swersky, Snoek, and Adams 2013; Yogatama and Mann 2014) address the knowledge transfer directly in the SMBO procedure.”
Recent Advances in Bayesian Optimization (2022, methodology): “25) has been modiied to improve the computational eiciency [122, 204].”
Learning search spaces for Bayesian optimization: Another view of hyperparameter transfer learning (2019, methodology): “Alternatively, several authors attempted to rely on manually deﬁned meta-features in order to measure the similarity between BO problems [3, 51, 35].”
Automated Machine Learning: Methods, Systems, Challenges (2019, unknown): “One can then learn how to combine these per-task predictions to warm-start or guide optimization techniques on a new task tnew [45, 114, 161, 187], as discussed in Sect.”
Inductive Transfer for Neural Architecture Optimization (2019, methodology): “Most of these methods also rely on metafeatures, i.e. features describing the properties of the dataset which are often not deﬁned for unstructured data (Yogatama & Mann, 2014; Schilling et al., 2016).”
Learning hyperparameter optimization initializations (2015, methodology): “…to H 6: λ * ← arg max λ * ∈Λ a (λ * , Ψ ) 7: Evaluate f (λ * ) 8: H ← H ∪ {(λ * , f (λ * ))} 9: return arg min (λ * ,f (λ * ))∈H f (λ * ) C. Gaussian Processes Gaussian processes [21] are a conventional prediction model for state of the art hyperparameter selection strategies [9], [5], [13], [10].”
Hyperparameter Search Space Pruning - A New Component for Sequential Model-Based Hyperparameter Optimization (2015, methodology): “Gaussian Process with MKL (MKL-GP) Similarly to Bardenet et al. [1], Yogatama and Mann [29] propose to use a Gaussian process as a surrogate model for the SMBO framework.”
Efficient and Robust Automated Machine Learning (2015, unknown): “Likewise, learning across datasets has also been applied in collaborative Bayesian optimization methods [16, 17]; while these approaches are promising, they are so far limited to very few meta-features and cannot yet cope with the highdimensional partially discrete configuration spaces faced in Auto”
Auto-sklearn: Efficient and Robust Automated Machine Learning (2019, methodology): “Likewise, learning across datasets has also been applied in collaborative Bayesian optimization methods [3, 38]; while these approaches are promising, they are so far limited to very few meta-features and cannot yet cope with the high-dimensional partially discrete conﬁguration spaces faced in…”
Taking Human out of Learning Applications: A Survey on Automated Machine Learning (2018, methodology): “by transferring the surrogate model [128] or its components such as kernel function [129].”
Bayesian Optimization of Text Representations (2015, background): “Other models, can be considered, of course, as can ensembles (Yogatama and Mann, 2014).”
Warmstarting of Model-based Algorithm Configuration (2017, background): “A prominent approach is to learn surrogate models across datasets (Swersky, Snoek, and Adams 2013; Bardenet et al. 2014; Yogatama and Mann 2014).”
Using Meta-Learning to Initialize Bayesian Optimization of Hyperparameters (2014, methodology): “Finally, we plan to integrate metalearning into the SMBO procedure and compare the result with recent work on collaborative SMBO [1, 35, 32].”