CMA-ES for Hyperparameter Optimization of Deep Neural Networks

By Ilya Loshchilov, Frank Hutter. 2016

240 citations recorded by the source corpus.

Hyperparameters of deep neural networks are often optimized by grid search, random search or Bayesian optimization. As an alternative, we propose to use the Covariance Matrix Adaptation Evolution Strategy (CMA-ES), which is known for its state-of-the-art performance in derivative-free optimization. CMA-ES has some useful invariance properties and is friendly to parallel evaluations of solutions. We provide a toy example comparing CMA-ES and state-of-the-art Bayesian optimization algorithms for tuning the hyperparameters of a convolutional neural network for the MNIST dataset on 30 GPUs in parallel.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Neural Architecture Search & AutoML

Citations in the corpus

Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution (2018, background): “, 2016], and evolutionary strategies [Loshchilov and Hutter, 2016].”
A genetic programming approach to designing convolutional neural network architectures (2017, methodology): “Naturally, evolutionary algorithms have also been applied to hyperparameter optimization problems [20].”
Evolving Deep Neural Networks (2017, background): “ recently, a studies have started to emerge with the goal of optimizing DNNs. Due to limited computational resources, they have focused on speci•c parts of the design. For instance, Loshchilov et al. [33] used CMA-ES to optimize the hyperparametersofexistingDNNsobtainingstate-of-the-artresultsone.g.”
Born to Learn: the Inspiration, Progress, and Future of Evolved Plastic Artificial Neural Networks (2017, methodology): “An established evolutionary computation technique, the Covariance Matrix Adaptation Evolution Strategy (CMA-ES), was used in Loshchilov and Hutter (2016) to optimize the parameters of a deep network to learn to classify the MNIST dataset.”
Simple And Efficient Architecture Search for Convolutional Neural Networks (2017, methodology): “…line of research on automated methods for setting these hyperparameters, including, e.g., random search (Bergstra & Bengio, 2012), Bayesian optimization (Bergstra et al., 2011; Snoek et al., 2012), bandit-based approaches (Li et al., 2016a), and evolutionary strategies (Loshchilov & Hutter, 2016).”
Evolutionary architecture search for deep multitask networks (2018, methodology): “us and difficult; as a result, much research focuses on developing automated methods for doing it. Some promising methods for hyperparameter search include deep Bayesian optimization [42], and CMA-ES [25]. One unique approach by Zoph and Le[52] uses reinforcement learning to develop an LSTM policy f”
SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization (2021, background): “In addition, evolutionary algorithms are also known as eﬃcient black-box optimizers (Fortin et al., 2012; Olson et al., 2016; Loshchilov and Hutter, 2016; Rapin and Teytaud, 2018; Awad et al., 2021).”
Evolving convolutional neural networks through grammatical evolution (2019, methodology): “The construction of CNN architectures can be expressed as a hyperparameter optimization problem [1, 2], where layers and layer parameters are all considered variables that can be optimized using search techniques, such as evolutionary algorithms [7, 9, 10].”
Hyperparameter Importance Across Datasets (2017, methodology): “hyperparameter fits a 1- dimensional Parzen Estimator to the values it took in Θbest. Similarly, as an analysis technique to study which values of a hyperparameter perform well, Loshchilov and Hutter [29] proposed to fit kernel density estimators (see, e.g., [33]) to these values. Here, we follow th”
Evolutionary neural AutoML for deep learning (2019, methodology): “CMA-ES has been shown to perform well in many real-world high-dimensional optimization problems and in particular, CMA-ES has been shown to outperform Bayesian optimization on tuning the parameters of a convolutional neural network [32].”
How to pick the domain randomization parameters for sim-to-real transfer of reinforcement learning policies? (2019, methodology): “We are interested in more advanced gradient-free optimization methods, such as CMA [34] or Bayesian optimization [35].”
Particle swarm optimization for hyper-parameter selection in deep neural networks (2017, methodology): “Other model-based techniques encompass the covariance matrix adaptation evolution strategies (CMA-ES) [28] and non-probabilistic methods, including RBF surrogate models [16].”
Multi-objective Architecture Search for CNNs (2018, background): “…setting hyperparameters of machine learning models, including, e.g., random search (Bergstra & Bengio, 2012), Bayesian optimization (Bergstra et al., 2011; Hutter et al., 2011; Snoek et al., 2012), bandit-based approaches (Li et al., 2016), and evolutionary strategies (Loshchilov & Hutter, 2016).”