Multi-objective Neural Architecture Search via Predictive Network Performance Optimization
By Han Shi, Renjie Pi, Hang Xu, Zhenguo Li, James T. Kwok, Tong Zhang. 2019
32 citations recorded by the source corpus.
Neural Architecture Search (NAS) has shown great potentials in finding a better neural network design than human design. Sample-based NAS is the most fundamental method aiming at exploring the search space and evaluating the most promising architecture. However, few works have focused on improving the sampling efficiency for a multi-objective NAS. Inspired by the nature of the graph structure of a neural network, we propose BOGCN-NAS, a NAS algorithm using Bayesian Optimization with Graph Convolutional Network (GCN) predictor. Specifically, we apply GCN as a surrogate model to adaptively discover and incorporate nodes structure to approximate the performance of the architecture. For NAS-oriented tasks, we also design a weighted loss focusing on architectures with high performance. Our method further considers an efficient multi-objective search which can be flexibly injected into any sample-based NAS pipelines to efficiently find the best speed/accuracy trade-off. Extensive experiments are conducted to verify the effectiveness of our method over many competing methods, e.g. 128.4x more efficient than Random Search and 7.8x more efficient than previous SOTA LaNAS for finding the best architecture on the largest NAS dataset NasBench-101.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Colin White
- Yash Savani
- Sam Nolen
- Willie Neiswanger
- Arber Zela
- Yuandong Tian
- Frank Hutter
- Linnan Wang
In Computer Vision and Pattern Recognition
- Colin White
- Yash Savani
- Luc Van Gool
- Sam Nolen
- Xiaogang Wang
- Jitendra Malik
- Dacheng Tao
- Trevor Darrell
Citations in the corpus
- BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search (2019, methodology): “…(Falkner, Klein, and Hutter 2018), NASBOT (Kandasamy et al. 2018b), local search (White, Nolen, and Savani 2020), TPE (Bergstra et al. 2011), BONAS (Shi et al. 2019), BOHAMIANN (Springenberg et al. 2016), REINFORCE (Williams 1992), GP-based BO (Snoek, Larochelle, and Adams 2012), AlphaX (Wang et…”
- Sample-Efficient Neural Architecture Search by Learning Action Space (2019, methodology): “Similarly, recent predictor-based methods [51] have achieved impressive results on NASBench by predicting every unseen architecture from the dataset.”
- Local Search is State of the Art for Neural Architecture Search Benchmarks (2020, methodology): “Popular techniques include Bayesian optimization [16, 13], reinforcement learning [40, 27, 20], gradient descent [21], neural prediction [36, 30], and evolution [28].”
- NAS-Bench-301 and the Case for Surrogate Benchmarks for Neural Architecture Search (2020, background): “Shi et al. (2019) and Wen et al. (2019) trained a Graph Convolutional Network (GCN) on a subset of NAS-Bench-101 (Ying et al., 2019) showing its effectiveness in predicting the performance of unseen architectures.”
- Latency-Aware Neural Architecture Search with Multi-Objective Bayesian Optimization (2021, background): “To address the NAS problem, the research community has developed a wide range of search algorithms, leveraging reinforcement learning (RL) (Zoph and Le, 2017; Tan et al., 2019), evolutionary search (ES) (Real et al., 2019; Liu et al., 2018a), and weight-sharing (Cai et al., 2020; Yu et al., 2020; Wa”