Efficient Neural Architecture Search via Proximal Iterations

By Quanming Yao, Ju Xu, Wei-Wei Tu, Zhanxing Zhu. 2019

108 citations recorded by the source corpus.

Neural architecture search (NAS) attracts much research attention because of its ability to identify better architectures than handcrafted ones. Recently, differentiable search methods become the state-of-the-arts on NAS, which can obtain high-performance architectures in several days. However, they still suffer from huge computation costs and inferior performance due to the construction of the supernet. In this paper, we propose an efficient NAS method based on proximal iterations (denoted as NASP). Different from previous works, NASP reformulates the search process as an optimization problem with a discrete constraint on architectures and a regularizer on model complexity. As the new objective is hard to solve, we further propose an efficient algorithm inspired by proximal iterations for optimization. In this way, NASP is not only much faster than existing differentiable search methods, but also can find better architectures and balance the model complexity. Finally, extensive experiments on various tasks demonstrate that NASP can obtain high-performance architectures with more than 10 times speedup over the state-of-the-arts.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Qiang Yang
2. Quanming Yao
3. James T. Kwok
4. Wei-Wei Tu
5. Tong Zhang
6. Yang Yu
7. Yi-Qi Hu
8. Yuhui Ding

In Computer Vision and Pattern Recognition

1. Quanming Yao
2. Tong Zhang
3. Yuhui Ding
4. Qiang Yang
5. Hugo Jair Escalante
6. Isabelle M Guyon
7. James T. Kwok
8. Huan Zhao

Citations in the corpus

• Generalizing from a Few Examples (2019, background): “For learning of deep networks, one-shot architecture search (OAS) is studied in [19, 83, 154].”
• Automated Machine Learning: From Principles to Practices (2018, methodology): “Sparse coding [35] is a representation learning method that aims at finding a sparse representation of the input data in the form of a linear combination of basic elements.”
• Optimal Transport Kernels for Sequential and Parallel Neural Architecture Search (2020, background): “…the increasing interest in deep learning in recent years, NAS has attracted signifi-cant research attention (Dong & Yang, 2019; Elsken et al., 2019a; Liu et al., 2018; 2019; Luo et al., 2018; Real et al., 2019; 2017; Shah et al., 2018; Suganuma et al., 2017; Xie & Yuille, 2017; Yao et al., 2020).”
• DiffMG: Differentiable Meta Graph Search for Heterogeneous Graph Neural Networks (2020, methodology): “Following the NAS framework [17, 30], at the search stage, we aim to solve a bi-level optimization problem:”
• Propagation Model Search for Graph Neural Networks (2020, methodology): “Recently, one-shot NAS methods (Cai, Zhu, and Han 2019; Dong and Yang 2019; Liu, Simonyan, and Yang 2019; Pham et al. 2018; Xie et al. 2019; Yao et al. 2020b) have been proposed to eliminate such expensive computation by training a single super network whose parameters are shared by all candidate ar”
• Simplifying Architecture Search for Graph Neural Network (2020, background): “Neural architecture search (NAS) [1, 6, 27, 32] aims to automatically find better and smaller architectures comparing to expert-designed ones, which have shown promising results in architecture design for CNN and Recurrent Neural Network (RNN) [17, 22, 28, 30, 33].”