Efficient Architecture Search by Network Transformation
By Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, Jun Wang. 2017
633 citations recorded by the source corpus.
Techniques for automatically designing deep neural network architectures such as reinforcement learning based approaches have recently shown promising results. However, their success is based on vast computational resources (e.g. hundreds of GPUs), making them difficult to be widely used. A noticeable limitation is that they still design and train each network from scratch during the exploration of the architecture space, which is highly inefficient. In this paper, we propose a new framework toward efficient architecture search by exploring the architecture space based on the current network and reusing its weights. We employ a reinforcement learning agent as the meta-controller, whose action is to grow the network depth or layer width with function-preserving transformations. As such, the previously validated networks can be reused for further exploration, thus saves a large amount of computational cost. We apply our method to explore the architecture space of the plain convolutional neural networks (no skip-connections, branching etc.) on image benchmark datasets (CIFAR-10, SVHN) with restricted computational resources (5 GPUs). Our method can design highly competitive networks that outperform existing networks using the same design scheme. On CIFAR-10, our model without skip-connections achieves 4.23% test error rate, exceeding a vast majority of modern architectures and approaching DenseNet. Furthermore, by applying our method to explore the DenseNet architecture space, we are able to achieve more accurate networks with fewer parameters.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neural Architecture Search & AutoML
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Mohit Bansal
- Yoshua Bengio
- Alan L. Yuille
- Jianfeng Gao
- Dacheng Tao
- Yi Yang
- Trevor Darrell
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Yi Yang
- Ming-Hsuan Yang
- Alan L. Yuille
- Xiaogang Wang
- Trevor Darrell
- Jitendra Malik
In Neural Architecture Search & AutoML
Citations in the corpus
- Progressive Neural Architecture Search (2017, methodology): “[7] use policy gradient to train an RNN, but the actions are now to widen an existing layer, or to deepen the network by adding an extra layer.”
- Regularized Evolution for Image Classifier Architecture Search (2018, background): “2015), parallelization, reward shaping and early stopping (Zhong, Yan, and Liu 2018) or Net2Net transformations (Cai et al. 2018).”
- Efficient Neural Architecture Search via Parameter Sharing (2018, methodology): “Neural architecture search (NAS) has been successfully applied to design model architectures for image classification and language models (Zoph & Le, 2017; Zoph et al., 2018; Cai et al., 2018; Liu et al., 2017; 2018).”
- DARTS: Differentiable Architecture Search (2018, background): “, 2018) and weight sharing/inheritance across multiple architectures (Elsken et al., 2017; Pham et al., 2018b; Cai et al., 2018; Bender et al., 2018), but the fundamental challenge of scalability remains.”
- AutoML: A Survey of the State-of-the-Art (2019, background): “4) Network Morphism based structure: Network Morphism (NM) based structure [13], [14], [20], [92] is the way of transferring the information stored in an existing neural network into a new neural network.”
- Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution (2018, methodology): “(2018) (ENAS) and Cai et al. (2018b) (PLNT), as well as with the multi-objective method by Dong et al.”
- A Comprehensive Survey of Neural Architecture Search (2020, methodology): “For example: EAS [50], Path-level EAS [56] and N2N learning [51], etc.”
- Fast Neural Network Adaptation via Parameter Remapping and Architecture Search (2020, methodology): “Following this manner, EAS (Cai et al., 2018) extends the parameter remapping concept to neural architecture search.”
- Understanding and Robustifying Differentiable Architecture Search (2019, methodology): “As these methods are often very expensive, various work focuses on reducing the search costs by, e.g., employing network morphisms [Cai et al., 2018a,b, Elsken et al., 2017, 2019b], weight sharing within one-shot models [Saxena and Verbeek, 2016, Bender et al., 2018, Pham et al., 2018] or…”
- Once for All: Train One Network and Specialize it for Efficient Deployment (2019, background): “Neural architecture search (NAS) focuses on automating the architecture design process [1, 16, 22, 32, 33].”
- PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search (2019, methodology): “Early efforts include sharing weights between searched and newly generated networks (Cai et al., 2018), and later these methods were generalized into a more elegant framework named one-shot architecture search (Brock et al.”
- Evolving Robust Neural Architectures to Defend from Adversarial Attacks (2019, methodology): “Some widely used search strategies for AAS are: Random Search, Bayesian Optimization [15], Evolutionary Methods [29],[41],[17],[28], Reinforcement Learning [44], [3], [6], [43],[7], [27] and Gradient Based Methods [4], [19], [18].”
- Automated Machine Learning: State-of-The-Art and Open Challenges (2019, methodology): “Neural Architecture Search (NAS) is a fundamental step in automating the machine learning process and has been successfully used to design the model architecture for image and language tasks [136, 137, 21, 73, 74].”
- Searching for a Robust Neural Architecture in Four GPU Hours (2019, methodology): “[5] reused weights of previously discovered networks to amortize the training cost.”
- Network Pruning via Transformable Architecture Search (2019, methodology): “Network transformation [5, 10, 3] also studied the depth and width of networks.”