AutoML-Zero: Evolving Machine Learning Algorithms From Scratch
By Esteban Real, Chen Liang, David R. So, Quoc V. Le. 2020
280 citations recorded by the source corpus.
Machine learning research has advanced in multiple aspects, including model structures and learning methods. The effort to automate such research, known as AutoML, has also made significant progress. However, this progress has largely focused on the architecture of neural networks, where it has relied on sophisticated expert-designed layers as building blocks---or similarly restrictive search spaces. Our goal is to show that AutoML can go further: it is possible today to automatically discover complete machine learning algorithms just using basic mathematical operations as building blocks. We demonstrate this by introducing a novel framework that significantly reduces human bias through a generic search space. Despite the vastness of this space, evolutionary search can still discover two-layer neural networks trained by backpropagation. These simple neural networks can then be surpassed by evolving directly on tasks of interest, e.g. CIFAR-10 variants, where modern techniques emerge in the top algorithms, such as bilinear interactions, normalized gradients, and weight averaging. Moreover, evolution adapts algorithms to different task types: e.g., dropout-like techniques appear when little data is available. We believe these preliminary successes in discovering machine learning algorithms from scratch indicate a promising new direction for the field.
Subfields: Artificial Intelligence, Neural Architecture Search & AutoML, Evolutionary Computation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Mohit Bansal
- Yoshua Bengio
- Sergey Levine
- Jianfeng Gao
- Percy Liang
- Graham Neubig
- Ruslan Salakhutdinov
In Neural Architecture Search & AutoML
In Evolutionary Computation
- Chen Liang
- Satinder Singh
- Andrew Brock
- Hanxiao Liu
- Quoc V. Le
- Esteban Real
- Zhangyang Wang
- Karen Simonyan
Citations in the corpus
- Neurosymbolic Programming (2021, background): “Recent work by Real et al. (2020) significantly extends the mandate of neural architecture search and brings it closer to general program synthesis.”
- Symbolic Discovery of Optimization Algorithms (2023, methodology): “Our approach builds upon a symbolic search space similar to AutoML-Zero [70, 80].”
- Learning to Optimize: A Primer and A Benchmark (2021, background): “Their idea was further developed in [174] to automatically discover complete machine learning algorithms from basic mathematical operations as building blocks, which concerns a more general problem than L2O.”
- Bootstrapped Meta-Learning (2021, background): “More generally, the update rule can be an arbitrary parameterised function (Hochreiter et al., 2001; Andrychowicz et al., 2016; Kirsch et al., 2019; Oh et al., 2020), or the function itself can be meta-learned jointly with its parameters (Alet et al., 2020; Real et al., 2020).”
- Evolving Normalization-Activation Layers (2020, background): “While existing works aim to specialize an architecture built upon well-defined building blocks such as Conv-BNReLU, we aim to find a universal layer that works well when paired with many different architectures, starting from lowlevel primitives similar to AutoML-Zero (Real et al., 2020).”