BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search
By Colin White, Willie Neiswanger, Yash Savani. 2019
413 citations recorded by the source corpus.
Over the past half-decade, many methods have been considered for neural architecture search (NAS). Bayesian optimization (BO), which has long had success in hyperparameter optimization, has recently emerged as a very promising strategy for NAS when it is coupled with a neural predictor. Recent work has proposed different instantiations of this framework, for example, using Bayesian neural networks or graph convolutional networks as the predictive model within BO. However, the analyses in these papers often focus on the full-fledged NAS algorithm, so it is difficult to tell which individual components of the framework lead to the best performance. In this work, we give a thorough analysis of the "BO + neural predictor framework" by identifying five main components: the architecture encoding, neural predictor, uncertainty calibration method, acquisition function, and acquisition function optimization. We test several different methods for each component and also develop a novel path-based encoding scheme for neural architectures, which we show theoretically and empirically scales better than other encodings. Using all of our analyses, we develop a final algorithm called BANANAS, which achieves state-of-the-art performance on NAS search spaces. We adhere to the NAS research checklist (Lindauer and Hutter 2019) to facilitate best practices, and our code is available at https://github.com/naszilla/naszilla.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neural Architecture Search & AutoML
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Colin White
- Yash Savani
- Frank Hutter
- Sam Nolen
- Willie Neiswanger
- Anne-Laure Boulesteix
- Zhenguo Li
- Marius Thomas Lindauer
In Computer Vision and Pattern Recognition
In Neural Architecture Search & AutoML
- Colin White
- Yash Savani
- Sam Nolen
- Marius Thomas Lindauer
- Willie Neiswanger
- Arber Zela
- Michael A. Osborne
- Swarnim Narayan
Citations in the corpus
- Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges (2021, methodology): “Among the possible optimization methods are: iterated local search (as used by Hutter et al. (2009)), evolutionary algorithms (as in White et al. (2021)), ES using derivatives (as used by Sekhon and Mebane (1998) and Roustant et al. (2012)), and a focusing RS called DIRECT (Jones, 2009).”
- Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS (2020, methodology): “In sample-based NAS algorithms, an architecture is selected in each iteration for full training [25, 33, 35, 42], and is computationally expensive.”
- Does Unsupervised Architecture Representation Learning Help Neural Architecture Search? (2020, methodology): “Similar to [43], we set the computational budget to 100 queries in this search space.”
- Optimal Transport Kernels for Sequential and Parallel Neural Architecture Search (2020, methodology): “We compare our approach against common baselines including Random search, evolutionary search, TPE (Bergstra et al., 2011), BOHB (Falkner et al., 2018), Nasbot (Kan-dasamy et al., 2018) and Bananas (White et al., 2021).”
- Local Search is State of the Art for NAS Benchmarks (2020, methodology): “For NASBOT, which was designed for macro (non cell-based) NAS, we computed the distance between two cells by taking the earth-mover’s distance between the set of row sums, column sums, and node operations, similar to (White et al., 2019).”
- Local Search is State of the Art for Neural Architecture Search Benchmarks (2020, methodology): “For NASBOT, which was designed for macro (non cell-based) NAS, we computed the distance between two cells by taking the earth-mover’s distance between the set of row sums, column sums, and node operations, similar to [36].”
- NAS-Bench-301 and the Case for Surrogate Benchmarks for Neural Architecture Search (2020, methodology): “An interesting alternative route to represent neural architectures was taken by BANANAS (White et al., 2019) which introduced a path-based encoding of cells that automatically resolves the computational equivalence of two architectures and used it to train a performance predictor to guide Bayesian…”
- Neural Architecture Search with Reinforce and Masked Attention Autoregressive Density Estimators (2020, methodology): “, 2019) when given an exploration budget of 3,200 architectures and significantly outperforms all algorithms with the exception of BANANAS (White et al., 2019).”
- A Study on Encodings for Neural Architecture Search (2020, methodology): “path encoding is a popular choice for neural predictor methods [22, 20, 18], and it was shown that truncating the path encoding leads to a small information loss [22].”
- Latency-Aware Neural Architecture Search with Multi-Objective Bayesian Optimization (2021, background): “, 2021), but only recently emerged as a promising approach for NAS (White et al., 2019; Liu et al., 2018b; Falkner et al., 2018; Kandasamy et al., 2018; Shi et al., 2019; Li and Talwalkar, 2020; Wang et al., 2019).”