Accelerating Neural Architecture Search using Performance Prediction

By Bowen Baker, Otkrist Gupta, Ramesh Raskar, Nikhil Naik. 2017

390 citations recorded by the source corpus.

Subfields: Efficiency & Compression, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

1. Xuanyi Dong
2. Yi Yang
3. Quoc V. Le
4. Barret Zoph
5. Wujie Wen
6. Hanxiao Liu
7. Xu Tan
8. Tao Qin

In Neural Architecture Search & AutoML

1. Frank Hutter
2. Xuanyi Dong
3. Colin White
4. Yash Savani
5. Quoc V. Le
6. Arber Zela
7. Xiaojun Chang
8. Quanming Yao

Citations in the corpus

• SNAS: Stochastic Neural Architecture Search (2018, background): “Ideas to design auxiliary mechanisms like performance prediction (Baker et al., 2017; Deng et al., 2017), iterative search (Liu et al., 2017a), hypernetwork generated weights (Brock et al., 2017) successfully accelerate NAS to certain degrees.”
• Neural Architecture Optimization (2018, background): “ccuracy on the dev set) of an architecture (the yellow surface in the middle part of Fig. 1). It is noteworthy here that the regression model is similar to the performance predictor in previous works [4, 27, 11]. What distinguishes our method is how to leverage the performance predictor: different w”
• Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution (2018, methodology): “Neural Architecture Search Recently, NAS was framed as a reinforcement learning (RL) problem, where the reward of the RL agent is based on the validation performance of the trained architecture (Baker et al., 2017a; Zoph & Le, 2017; Zhong et al., 2018; Pham et al., 2018).”
• A Comprehensive Survey of Neural Architecture Search (2020, methodology): “Therefore, a lot of work has emerged on how to reduce the amount of calculation and accelerate the search of the network architecture [18–20, 48, 49, 52, 84, 105].”
• Surrogate-Assisted Evolutionary Deep Learning Using an End-to-End Random Forest-Based Performance Predictor (2020, methodology): “In addition, the train-less accuracy predictor (TAP) [34] and the accelerating neural network architecture search (ANNAS) [35] algorithms also employed similar methods to predict the performance of the given CNNs.”
• Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, positive): “Meta learning is also applicable for directly predicting performance of a given model with certain hyper-parameter configurations, by building a surrogate model with previous configurations (Eggensperger et al., 2015; Baker et al., 2017).”
• NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search (2020, methodology): “…insights for better designs and training strategies of different NAS algorithms, such as platform-aware NAS (Tan et al., 2019), accuracy prediction (Baker et al., 2018), mutationbased NAS (Cai et al., 2018; Chen et al., 2016), etc. Architecture Computational Costs: NAS-Bench-201 provides three…”
• BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search (2019, background): “There are several works that predict the validation accuracy of neural networks (Klein et al. 2017; Deng, Yan, and Lin 2017; Istrate et al. 2019; Zhang, Ren, and Urtasun 2018; Baker et al. 2017).”
• Understanding and Robustifying Differentiable Architecture Search (2019, background): “…2019b], weight sharing within one-shot models [Saxena and Verbeek, 2016, Bender et al., 2018, Pham et al., 2018] or multi-fidelity optimization [Baker et al., 2017b, Falkner et al., 2018, Li et al., 2017, Zela et al., 2018], however their applicability still often remains restricted to rather…”
• Searching for a Robust Neural Architecture in Four GPU Hours (2019, methodology): “A variety of techniques have been proposed, such as progressivecomplexity search stages [22], accuracy prediction [2], HyperNet [4], Net2Net transformation [5], and parameter sharing [31].”
• A Survey on Neural Architecture Search (2019, methodology): “ror between all mature learning curves and a premature learning curve. The prediction model is formed as an average of the best ts of mature learning curves. 30 A Survey on Neural Architecture Search Baker et al. (2018) All previously discussed learning curve predictors are based only on the prematu”
• Graph HyperNetworks for Neural Architecture Search (2018, methodology): “Baker et al. (2017b) propose directly predicting performance from the learning curve, and Deng et al. (2017) propose to predict performance directly from the architecture without learning curve information.”
• Auto-Meta: Automated Gradient Based Meta Learner Search (2018, background): “This line of thinking is present in hypernetworks [4], co-evolving NEAT [15], boosting [6, 11], MCTS [17], early stopping of unpromising models [3], and progressive neural architecture search(PNAS) [13].”
• TAPAS: Train-less Accuracy Predictor for Architecture Search (2018, methodology): “2017), ν-SVR (Baker et al. 2018) forecast network performance based on partial learning curves.”
• Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting (2019, methodology): “Our work also relates to neural architecture search (Stanley & Miikkulainen, 2002; Zoph & Le, 2016; Baker et al., 2017; Liu et al., 2018), as we employ search methods for the implementation of structure optimization.”
• DPP-Net: Device-aware Progressive Search for Pareto-optimal Neural Architectures (2018, background): “[24] proposed to predict performance to reduce the effort of searching model architectures.”
• ChamNet: Towards Efficient Network Design Through Platform-Aware Model Adaptation (2018, neutral): “NAS and SMBO: Platform-aware NAS and SMBO have emerged as a promising direction for automating the synthesis flow of a model based on direct metrics, making it more suitable for deployment [14, 22, 3, 19, 20, 4].”