Towards Automatically-Tuned Neural Networks

By Hector Mendoza, Aaron Klein, Matthias Feurer, Jost Tobias Springenberg, Frank Hutter. 2016

222 citations recorded by the source corpus.

Subfields: Optimization & Training, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

In Neural Architecture Search & AutoML

Citations in the corpus

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, methodology): “Finally, previous work has explored building ensembles of models produced by hyperparameter searches (Snoek et al., 2015; Mendoza et al., 2016; Saikia et al., 2020), including greedy selection strategies (Caruana et al.”
Neural Architecture Search with Reinforcement Learning (2016, methodology): “d with a good initial model (Bergstra & Bengio, 2012; Snoek et al., 2012; 2015). There are Bayesian optimization methods that allow to search non ﬁxed length architectures (Bergstra et al., 2013; Mendoza et al., 2016), but they are less general and less ﬂexible than the method proposed in this p”
Learning Transferable Architectures for Scalable Image Recognition (2017, methodology): “rovide a viable search strategy, RL does signiﬁcantly improve our ability to learn neural architectures. 4. Related Work The proposed method is related to previous work in hyperparameter optimization [43, 4, 5, 53, 54, 6, 39] – especially recent approaches in designing architectures such as Neural F”
Progressive Neural Architecture Search (2017, methodology): “, (Mendoza et al., 2016; Negrinho & Gordon, 2017)), but none have shown the kinds of gains on large scale image classification problems that we do.”
Regularized Evolution for Image Classifier Architecture Search (2018, background): “Other methods have also been applied: cascade-correlation [16], boosting [10], hill-climbing [14], MCTS [33], SMBO [28, 30], and random search [4], and grid search [49].”
AutoML: A Survey of the State-of-the-Art (2019, methodology): “The widely used methods contains reinforcement learning (RL) [5], [6], [8], [9], [16], evolutionary algorithm (EA) [17]–[23], and gradient descent (GD) [10], [24], [25], Bayesian optimization [26]– [32] and so on.”
BOHB: Robust and Efficient Hyperparameter Optimization at Scale (2018, methodology): “We used the Bayesian neural network implementation provided in the RoBO python package (Klein et al., 2017b) as described by Springenberg et al. (2016).”
Differentiable Neural Input Search for Recommender Systems (2020, methodology): “twork architecture. To explore the space of neural architectures, different search strategies have been explored including random search [22], evolutionary methods [12, 29, 34], Bayesian optimization [2, 11, 28], reinforcement learning [1, 40, 41], and gradient-based methods [3, 25, 38]. Since being”
Automated Machine Learning: State-of-The-Art and Open Challenges (2019, methodology): “Auto-Net [82] is an efficient neural architecture search framework based on SMAC optimization and built on top of PyTorch.”
sharpDARTS: Faster and More Accurate Differentiable Architecture Search (2019, neutral): “Furthermore, once a search space is de-ﬁned various tools can be brought to bear on the problem including Bayesian optimization[16], other neural networks[1], reinforcement learning, evolution[21, 20], or a wide variety of optimization frameworks.”
Bayesian Learning of Neural Network Architectures (2019, methodology): “…network architecture search has long been a topic of research and diverse methods such as evolutionary algorithms [Todd, 1988, Miller et al., 1989, Kitano, 1990], reinforcement learning [Zoph and Le, 2016] or Bayesian optimisation [Bergstra et al., 2013, Mendoza et al., 2016] have been applied.”
Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search (2018, background): “While most Bayesian optimization methods only focus on a few continuous hyperparameters, ? and Mendoza et al. (2016) optimized architecture and hyperparameters jointly and achieved, at this time, state-of-the-art performance for shallow convolutional neural networks and feed forward neural…”
Path-Level Network Transformation for Efficient Architecture Search (2018, background): “, 2018), Bayesian optimization (Domhan et al., 2015; Mendoza et al., 2016), Monte Carlo Tree Search (Negrinho & Gordon, 2017) and reinforcement learning (RL) (Zoph & Le, 2017; Baker et al.”
Neural Architecture Search with Bayesian Optimisation and Optimal Transport (2018, methodology): “While there has been some work on BO for architecture search [2, 15, 25, 37, 41], they have only been applied to optimise feed forward structures, e.”
Efficient Architecture Search by Network Transformation (2017, methodology): “In parallel, automatic architecture designing has also been studied in the context of Bayesian optimization (Bergstra and Bengio 2012; Domhan, Springenberg, and Hutter 2015; Mendoza et al. 2016).”
Simple And Efficient Architecture Search for Convolutional Neural Networks (2017, background): “While architectural choices can be treated as categorical hyperparameters and be optimized with standard hyperparameter optimization methods (Bergstra et al., 2011; Mendoza et al., 2016), the current focus is on the development of special techniques for architectural optimization.”
A brief Review of the ChaLearn AutoML Challenge: Any-time Any-dataset Learning without Human Intervention (2016, methodology): “For the GPU version (Mendoza et al., 2016), they used the Java version of SMAC to tune AutoSklearn and deep neural networks implemented in Lasagne/Theano (Dieleman et al.”
Hyperparameter Ensembles for Robustness and Uncertainty Quantification (2020, background): “This idea has been made systematic as part of auto-sklearn [18] and successfully exploited in several other contexts, e.g., [16] and speciﬁcally for neural networks [43].”