Neural Architecture Optimization

By Renqian Luo, Fei Tian, Tao Qin, Tie-Yan Liu. 2018

720 citations recorded by the source corpus.

Automatic neural architecture design has shown its potential in discovering powerful neural network architectures. Existing methods, no matter based on reinforcement learning or evolutionary algorithms (EA), conduct architecture search in a discrete space, which is highly inefficient. In this paper, we propose a simple and efficient method to automatic neural architecture design based on continuous optimization. We call this new approach neural architecture optimization (NAO). There are three key components in our proposed approach: (1) An encoder embeds/maps neural network architectures into a continuous space. (2) A predictor takes the continuous representation of a network as input and predicts its accuracy. (3) A decoder maps a continuous representation of a network back to its architecture. The performance predictor and the encoder enable us to perform gradient based optimization in the continuous space to find the embedding of a new architecture with potentially better accuracy. Such a better embedding is then decoded to a network by the decoder. Experiments show that the architecture discovered by our method is very competitive for image classification task on CIFAR-10 and language modeling task on PTB, outperforming or on par with the best results of previous architecture search methods with a significantly reduction of computational resources. Specifically we obtain $2.11\%$ test set error rate for CIFAR-10 image classification task and $56.0$ test set perplexity of PTB language modeling task. The best discovered architectures on both tasks are successfully transferred to other tasks such as CIFAR-100 and WikiText-2. Furthermore, combined with the recent proposed weight sharing mechanism, we discover powerful architecture on CIFAR-10 (with error rate $3.53\%$) and on PTB (with test set perplexity $56.6$), with very limited computational resources (less than $10$ GPU hours) for both tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Microsoft Research Asia’s Systems for WMT19 (2019, methodology): “NAO (Luo et al., 2018) is a gradient based neural architecture search (NAS) method.”
On Network Design Spaces for Visual Recognition (2019, neutral): “Most work on NAS focuses on the search algorithm, and various search strategies have been studied, including RL [38, 39, 28], heuristic search [20], gradient-based search [21, 23], and evolutionary algorithms [29].”
MnasNet: Platform-Aware Neural Architecture Search for Mobile (2018, background): “These approaches are mainly based on reinforcement learning [35, 36, 1, 19, 25], evolutionary search [26], differentiable search [21], or other learning algorithms [19, 17, 23].”
Advances and Open Problems in Federated Learning (2019, background): “He, M. Annavaram, and S. Avestimehr, “FedNAS: Federated deep learning via neural architecture search,” arXiv preprint arXiv:2004.08546, 2020.”
MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulation (2020, methodology): “The third method is gradient-based [17, 28, 5, 19, 4], which relaxes the categorical design choices to continuous variables and then leverages the efficient gradient backpropagation so that it can finish searching within as little as several GPU days.”
A Comprehensive Survey of Neural Architecture Search (2020, methodology): “s process until a sufficient number of blocks of cells are found. As shown by the black curve in Fig.14, we show an example of such performance prediction based on early performance observations. NAO [73] uses a performance predictor similar to previous work [20, 36, 109]. Unlike PNAS [36] that uses”
AutoEmb: Automated Embedding Dimensionality Search in Streaming Recommendations (2020, background): “Luo et al.[18] reflects the neural architectures into an embedding space so that the optimal embedding could be learned and given as feedback to the final architecture.”
Semi-Supervised Neural Architecture Search (2020, methodology): “Conventional NAS includes [41, 42, 20, 17], which achieve significant improvements on several benchmark datasets.”
On Robustness of Neural Architecture Search Under Label Noise (2020, methodology): “…optimization (Kandasamy et al., 2018; Jin et al., 2019), reinforcement learning (Zoph and Le, 2017; Cai et al., 2018; Pham et al., 2018; Zoph et al., 2018), evolutionary algorithms (Real et al., 2017, 2019) and gradient-based optimization (Luo et al., 2018; Cai et al., 2019; Liu et al., 2019).”
NAS evaluation is frustratingly hard (2019, methodology): “They ﬁnd that the search spaces of three methods investigated (DARTS, ENAS, NAO) have been expertly engineered to the extent that any randomly selected architecture performs very well.”
DARTS+: Improved Differentiable Architecture Search with Early Stopping (2019, methodology): “The neural architectures searched by NAS have achieved the state-of-the-art results over handcrafted neural architectures in various tasks, including object classiﬁcation [4, 20, 21, 25, 26, 40], object detection [7, 36], semantic segmentation [19], recommender systems [14], etc.”
Auto-GNN: Neural architecture search of graph neural networks (2019, methodology): “The deep architectures discovered by NAS algorithms have outperformed the handcrafted ones at the domains including image classification [12–21], semantic image segmentation [22], and image generation [23].”
PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search (2019, neutral): “To this end, the trainnig set is partitioned into two parts, with the ﬁrst part used for optimizing network parameters, e.g. , (Real et al., 2019) 2.55 ± 0.05 2.8 3150 evolution Hireachical Evolution (Liu et al., 2018b) 3.75 ± 0.12 15.7 300 evolution PNAS (Liu et al., 2018a) 3.41 ± 0.09 3.2 225 SMBO”
FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search (2019, background): “While it seems ideal, most of the studies on neural architecture search behave incredibly poorly with this metric [14, 15], with the best reported τ be 0.”
Evolving Robust Neural Architectures to Defend from Adversarial Attacks (2019, background): “arch strategies for AAS are: Random Search, Bayesian Optimization [15], Evolutionary Methods [29],[41],[17],[28], Reinforcement Learning [44], [3], [6], [43],[7], [27] and Gradient Based Methods [4], [19], [18]. Finally, a performance estimation (usually error rate) is required to evaluate the explo”
Automated Machine Learning: State-of-The-Art and Open Challenges (2019, background): “NAO [77], and Darts [75] are open source frameworks for neural architecture search which propose a new continuous optimization algorithm that deals with the network architecture as a continuous space instead of the discretization followed by other approaches.”
Searching for a Robust Neural Architecture in Four GPU Hours (2019, background): “Recently, neural architecture search (NAS) has increasingly attracted the interest of researchers [1, 6, 17, 22, 25, 31, 46].”
EENA: Efficient Evolution of Neural Architecture (2019, methodology): “[26] use an encoder network mapping neural network architectures into a continuous space.”