Multi-Task Bayesian Optimization

By Kevin Swersky, Jasper Snoek, Ryan P. Adams. 2013

842 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Learning to Optimize Neural Nets (2017, methodology): “…kinds of methods have been proposed, such those based on Bayesian optimization (Hutter et al., 2011; Bergstra et al., 2011; Snoek et al., 2012; Swersky et al., 2013; Feurer et al., 2015), random search (Bergstra & Bengio, 2012) and gradient-based optimization (Bengio, 2000; Domke, 2012;…”
Self-Supervised Learning of Visual Features through Embedding Images into Text Topic Spaces (2017, neutral): “[34] * 70.10% C-SVDDNet [37] 68.23% K-means (Single layer net) [4] 51.50% Raw pixels 31.80% and compute the KL-divergence distance of the query (image or text) with all the entities in the database.”
Discriminative Unsupervised Feature Learning with Convolutional Neural Networks (2014, neutral): “1[30] — 91.”
Scattering Networks for Hybrid Representation Learning (2018, methodology): “The best reported result in the purely supervised case is a CNN [19], [52] whose hyper parameters have been automatically tuned using 4,000 images for validation achieving 70.”
Stacked What-Where Auto-encoders (2015, neutral): “9% Multi-task Bayesian Optimization [27] 70 .”
BOHB: Robust and Efficient Hyperparameter Optimization at Scale (2018, methodology): “For instance, multi-task Bayesian optimization (Swersky et al., 2013) exploits correlation between tasks to warm-start the optimization procedure.”
Scalable Bayesian Optimization Using Deep Neural Networks (2015, background): “…elegant theoretical results (Srinivas et al., 2010; Bull, 2011; de Fre-itas et al., 2012), multitask and transfer optimization (Krause & Ong, 2011; Swersky et al., 2013; Bar-denet et al., 2013) and the application to diverse tasks such as sensor set selection (Garnett et al., 2010), the tuning…”
Random Search and Reproducibility for Neural Architecture Search (2019, background): “Bayesian approaches based on Gaussian processes [43; 26; 44; 24] and gradient-based approaches [2; 35] are generally only applicable to continuous search spaces.”
Forward and Reverse Gradient-Based Hyperparameter Optimization (2017, background): “Copyright 2017 by the author(s). response function, Bayesian optimization approaches provide a natural framework and have been extensively studied in this context (Snoek et al., 2012; Swersky et al., 2013; Snoek et al., 2015).”
Designing Neural Network Architectures using Reinforcement Learning (2016, methodology): “Bayesian optimization has also been used (Shahriari et al., 2016) for automatic selection of network architectures (Bergstra et al., 2013; Domhan et al., 2015) and hyperparameters (Snoek et al., 2012; Swersky et al., 2013).”
Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets (2016, methodology): “Other approaches for hyperparameter optimization on large datasets include recent work by Nickson et al. [15], who estimated a conﬁguration’s performance on a large dataset by evaluating several training runs on small, random subsets of ﬁxed, manually-chosen sizes.”
Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization (2016, methodology): “Domhan et al. (2015); Swersky et al. (2013, 2014); Gy¨orgy and Kocsis (2011); Agarwal et al. (2011) propose methods that make parametric assumptions on the convergence behavior of training algorithms, providing theoretical performance guarantees under these assumptions.”
Taking the Human Out of the Loop: A Review of Bayesian Optimization (2016, background): “Returning to the drug application, we can imagine the effectiveness of different drugs (arms on the bandit) as being determined by a function f that takes an index a ∈ 1 , . . . , K and returns a Bernoulli parameter in the interval (0 , 1) .”
Learning to Optimize (2016, methodology): “Most methods [13, 4, 24, 25, 9] rely on sequential model-based Bayesian optimization [22, 6], while others adopt a random search approach [3] or use gradient-based optimization [2, 8, 21].”
Population Based Training of Neural Networks (2017, background): “…process and to better explore the space of promising hyperparameters (Gy¨orgy & Kocsis, 2011; Agarwal et al., 2011; Sabharwal et al., 2016; Swersky et al., 2013, 2014; Domhan et al., 2015; Klein et al., 2016; Snoek et al., 2015; Springenberg et al., 2016). runs to be completed…”
Learning to Learn for Global Optimization of Black Box Functions (2016, methodology): “Methods that use an auxiliary model to represent the learning process have the most potential for transfer, but may require additional work to extract the appropriate information from this model [Swersky et al., 2013].”
Convolutional Kernel Networks (2014, neutral): “t better results than those reported in Table 2 have been obtained in the literature by using either data augmentation (around 90% on CIFAR-10 for [18, 30]), or external data (around70% on STL-10 for [28]). We are planning to investigate similar data manipulations in the future. Method [12] [27] [18”
Freeze-Thaw Bayesian Optimization (2014, methodology): “Instead, similar to [18], we use the entropy search acquisition function to pick the point that provides the most information about the location of the minimum at the asymptote.”
Meta-Learning: A Survey (2018, neutral): “Earlier work on multi-task learning (Swersky et al., 2013) assumed that we already have a set of ‘similar’ source tasks tj .”
Initializing Bayesian Hyperparameter Optimization via Meta-Learning (2015, methodology): “These collaborative SMBO methods (Bardenet et al. 2013; Swersky, Snoek, and Adams 2013; Yogatama and Mann 2014) address the knowledge transfer directly in the SMBO procedure.”