Population Based Training of Neural Networks

By Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M. Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan and 2 more. 2017

896 citations recorded by the source corpus.

Neural networks dominate the modern machine learning landscape, but their training and success still suffer from sensitivity to empirical choices of hyperparameters such as model architecture, loss function, and optimisation algorithm. In this work we present \emph{Population Based Training (PBT)}, a simple asynchronous optimisation algorithm which effectively utilises a fixed computational budget to jointly optimise a population of models and their hyperparameters to maximise performance. Importantly, PBT discovers a schedule of hyperparameter settings rather than following the generally sub-optimal strategy of trying to find a single fixed set to use for the whole course of training. With just a small modification to a typical distributed hyperparameter training framework, our method allows robust and reliable training of models. We demonstrate the effectiveness of PBT on deep reinforcement learning problems, showing faster wall-clock convergence and higher final performance of agents by optimising over a suite of hyperparameters. In addition, we show the same method can be applied to supervised learning for machine translation, where PBT is used to maximise the BLEU score directly, and also to training of Generative Adversarial Networks to maximise the Inception score of generated images. In all cases PBT results in the automatic discovery of hyperparameter schedules and model selection which results in stable training and better final performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, methodology): “More generally, our findings may impact the design of automated methods for solving/optimizing machine learning problems (Feurer et al., 2015; Zoph & Le, 2017; Jaderberg et al., 2017).”
Learning and Evaluating General Linguistic Intelligence (2019, methodology): “Recent progress on methods such as Population Based Training (PBT, Jaderberg et al., 2017) could be used to design a curriculum that improves training.”
The challenge of realistic music generation: modelling raw audio at scale (2018, methodology): “Note that VQ-VAE models with larger decoder RFs do not converge, even with PBT. Table 3: Overview of the models we consider for qualitative evaluation, with ratings out of ﬁve for signal ﬁdelity and musicalty from an informal human evaluation.”
Off-Policy Actor-Critic with Shared Experience Replay (2019, background): “On DMLab-30 we consider both regular hyper-parameter sweeps and sweeps with population based training (PBT) (Jaderberg et al., 2017a).”
IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures (2018, methodology): “Since the current state-of-the-art methods like A3C (Mnih et al., 2016) or UNREAL (Jaderberg et al., 2017b) can require as much as a billion frames and multiple days to master a single domain, training them on tens of domains at once is too slow to be practical.”
Regularized Evolution for Image Classifier Architecture Search (2018, background): “, 2017) and some used evolution to train a single architecture (Jaderberg et al., 2017) or to find its weights (Such et al.”
Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, methodology): “Figure 16: PBT as a combination of sequential and parallel optimization (Jaderberg et al., 2017)”
AutoML-Zero: Evolving Machine Learning Algorithms From Scratch (2020, methodology): “AutoML has utilized a wide array of paradigms, including growing networks neuron-by-neuron (Stanley & Miikkulainen, 2002), hyperparameter optimization (Snoek et al., 2012; Loshchilov & Hutter, 2016; Jaderberg et al., 2017) and, neural architecture search (Zoph & Le, 2016; Real et al.”
Adversarial AutoAugment (2019, methodology): “Inspired by population based training (PBT) (Jaderberg et al., 2017), the augmentation policy search problem in PBA is modeled as a process of hyperparameter schedule learning.”
Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules (2019, methodology): “We describe the formulation of our search in the format of PBT experiments (Jaderberg et al., 2017).”
Fast AutoAugment (2019, background): “Recently, PBA [13] proposed a new algorithm which generates augmentation policy schedules based on population based training [15].”
Random Search and Reproducibility for Neural Architecture Search (2019, background): “, evolutionary search [21, 40], Bayesian optimization [4, 20, 44], and gradient-based approaches [2, 36].”
Meta-Gradient Reinforcement Learning (2018, methodology): “In principle, γ and λ could be exposed to population-based training (PBT) [Jaderberg et al., 2017a], however, this would blow up the computation time by the size of the population (24 in Espeholt et al. [2018]), which is beyond reach for typical experiments; furthermore adaptation by PBT does not…”
Fast Conditional Independence Test for Vector Variables with Large Sample Sizes (2018, background): “…small and large, high-and low-dimensional data, the hyperparameter choice for the neural network regressor becomes a complex problem – for example Jaderberg et al. [2017] propose an simple and effective hyperparameter search method that nevertheless assumes having access to a cluster of a…”
Towards Interpretable Reinforcement Learning Using Attention Augmented Agents (2019, methodology): “Following [33], we train the agent using population based training ([34]) with a population size of 16, where we evolve the learning rate, entropy cost, RMSProp (cid:15) , and gradient clipping threshold.”
Online Continual Learning with Natural Distribution Shifts: An Empirical Study with Visual Data (2021, methodology): “We also evaluate an adaptive schedule generated by populationbased search [14].”