Neural Optimizer Search with Reinforcement Learning

By Irwan Bello, Barret Zoph, Vijay Vasudevan, Quoc V. Le. 2017

408 citations recorded by the source corpus.

We present an approach to automate the process of discovering optimization methods, with a focus on deep learning architectures. We train a Recurrent Neural Network controller to generate a string in a domain specific language that describes a mathematical update equation based on a list of primitive functions, such as the gradient, running average of the gradient, etc. The controller is trained with Reinforcement Learning to maximize the performance of a model after a few epochs. On CIFAR-10, our method discovers several update rules that are better than many commonly used optimizers, such as Adam, RMSProp, or SGD with and without Momentum on a ConvNet model. We introduce two new optimizers, named PowerSign and AddSign, which we show transfer well and improve training on a variety of different tasks and architectures, including ImageNet classification and Google's neural machine translation system.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark (2019, methodology): “For example by learning: architectures Zoph & Le (2017), optimization algorithms Bello et al. (2017), initialization distributions Raghu et al. (2019), or pre-trained data representations.”
Attention Augmented Convolutional Networks (2019, background): “As such, it would be interesting to see if using Attention Augmentation as a primitive in automated architecture search procedures proves useful to find even better models than those previously found in image classification [55], object detection [12], image segmentation [6] and other domains[5, 1, ”
AutoAugment: Learning Augmentation Policies from Data (2018, methodology): “Search algorithm details: The search algorithm that we used in our experiment uses Reinforcement Learning, inspired by [19, 23, 32, 51].”
Regularized Evolution for Image Classifier Architecture Search (2018, neutral): “, 2017) and optimizers (Bello et al., 2017), for example.”
Efficient Neural Architecture Search via Parameter Sharing (2018, background): “0 for the sampling logits (Bello et al., 2017a;b), and add the controller’s sample entropy to the reward, weighted by 0 .”
Swish: a Self-Gated Activation Function (2017, methodology): “5% top-1 accuracy on the Mobile NASNet-A model, pushing the difference between Swish and ReLU to a sizeable 1 .”
Advances and Open Problems in Federated Learning (2019, background): “He, M. Annavaram, and S. Avestimehr, “FedNAS: Federated deep learning via neural architecture search,” arXiv preprint arXiv:2004.08546, 2020.”
Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, background): “To save the rare resource of experts experience, automated machine learning (AutoML) has been proposed as a burgeoning technology to design and train neural networks automatically, at the cost of computational resources (Feurer et al., 2015; Katz et al., 2016; Bello et al., 2017; Zoph and Le, 2016; ”
AutoML-Zero: Evolving Machine Learning Algorithms From Scratch (2020, methodology): “[7] revisited symbolically learned optimizers to apply them to a modern neural network.”
Searching for a Robust Neural Architecture in Four GPU Hours (2019, neutral): “Most NAS approaches apply evolutionary algorithms (EA) [32, 23, 33] or reinforcement learning (RL) [46, 47, 3] to design neural architectures automatically.”
A Survey on Neural Architecture Search (2019, methodology): “For instance, Bello et al. (2017) encode a parameter optimization method used in training of a deep learning model by means of a graph structure that defines the update rules akin to classical gradient-based optimization.”
Learning To Simulate (2018, neutral): “This concept is similar to recent papers that learn policies for neural network architectures (Zoph & Le, 2016) and optimizers (Bello et al., 2017).”
Evolutionary-Neural Hybrid Agents for Architecture Search (2018, methodology): “RL-based neural architecture search yielded success in automatic design of state-of-the-art RNN cells [50], convolutional blocks [51], activation functions [32], optimizers [3, 42] and data augmentation strategies [10].”
Transfer Learning with Neural AutoML (2018, methodology): “These include random search [1], parameter modeling [3], metalearned hyperparameter initialization [16], deep-learning based tree searches over a predefined model-specification language [17], and learning of gradient descent optimizers [18, 19].”
An overview of deep learning in medical imaging focusing on MRI (2018, methodology): “It was created using AutoML(23), Google Brain’s reinforcement learning approach to architecture design [78].”
Reinforced Continual Learning (2018, methodology): “It has been applied successfully to various problems, such as games [7, 10], natural language processing [16], neural architecture/optimizer search [18, 1] and so on.”
Meta-Learning in Neural Networks: A Survey (2020, methodology): “When it is computed as the sum of the validation loss after each inner optimization step, then meta-training also encourages faster learning in the base task [78], [91], [92].”