Efficient Neural Architecture Search via Parameter Sharing

By Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, Joseph Dean. 2018

3,045 citations recorded by the source corpus.

We propose Efficient Neural Architecture Search (ENAS), a fast and inexpensive approach for automatic model design. In ENAS, a controller learns to discover neural network architectures by searching for an optimal subgraph within a large computational graph. The controller is trained with policy gradient to select a subgraph that maximizes the expected reward on the validation set. Meanwhile the model corresponding to the selected subgraph is trained to minimize a canonical cross entropy loss. Thanks to parameter sharing between child models, ENAS is fast: it delivers strong empirical performances using much fewer GPU-hours than all existing automatic model design approaches, and notably, 1000x less expensive than standard Neural Architecture Search. On the Penn Treebank dataset, ENAS discovers a novel architecture that achieves a test perplexity of 55.8, establishing a new state-of-the-art among all methods without post-training processing. On the CIFAR-10 dataset, ENAS designs novel architectures that achieve a test error of 2.89%, which is on par with NASNet (Zoph et al., 2018), whose test error is 2.65%.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

AutoGrow: Automatic Layer Growing in Deep Convolutional Networks (2019, neutral): “To accelerate NAS, one-shot models [29, 26, 2], DARTS [22] and NAS with Transferable Cell [40, 20] were proposed.”
Pathways: Asynchronous Distributed Dataflow for ML (2022, background): “, 2020), neural architecture search (Pham et al., 2018), and multi-modal multi-task learning systems (Ma et al.”
Microsoft Research Asia’s Systems for WMT19 (2019, background): “To reduce computational cost, we share weights among different architectures (Pham et al., 2018) to aid the generation of such paired training data.”
Randaugment: Practical automated data augmentation with a reduced search space (2019, methodology): “Subsequent work accelerated training efﬁciency and the ef-ﬁcacy of the procedure [30, 38, 28, 29], eventually making the method amenable to a uniﬁed optimization based on a differentiable process [30].”
The Efficiency Misnomer (2021, methodology): “To this end, these studies mostly use the parameter size (Pham et al., 2018), FLOPs (Hsu et al.”
Why Should We Add Early Exits to Neural Networks? (2020, background): “Searching for efficient architectures is a wide problem in the literature on neural model selection [54].”
On Network Design Spaces for Visual Recognition (2019, methodology): “In our setting, training 500 CIFAR models requires about 250 GPU hours.”
Squeeze-and-Excitation Networks (2017, background): “Particularly strong results have been achieved with techniques from reinforcement learning [40]–[44].”
Progressive Neural Architecture Search (2017, methodology): “[15] use parameter sharing among child models to substantially accelerate the search process.”
DARTS: Differentiable Architecture Search (2018, methodology): “The very ﬁrst intermediate node is obtained by linearly transforming the two input nodes, adding up the results and then passing through a tanh activation function, as done in the ENAS cell (Pham et al., 2018b).”
The Evolved Transformer (2019, methodology): “, 2018b), ENAS (Pham et al., 2018), SMASH (Brock et al.”
Advances and Open Problems in Federated Learning (2019, background): “The recent progress in NAS [332, 154, 333, 55, 322, 273, 417, 154, 279] provides a potential way to address these drawbacks.”
SNAS: Stochastic Neural Architecture Search (2018, methodology): “Reinforcement-learning-based NAS is end-to-end for gradient back-propagation, among which the most efficient one, ENAS (Pham et al., 2018) learns optimal parameters and architectures together just like NEAT.”