SMASH: One-Shot Model Architecture Search through HyperNetworks
By Andrew Brock, Theodore Lim, James M. Ritchie, Nick Weston. 2017
808 citations recorded by the source corpus.
Designing architectures for deep neural networks requires expert knowledge and substantial computation time. We propose a technique to accelerate architecture selection by learning an auxiliary HyperNet that generates the weights of a main model conditioned on that model's architecture. By comparing the relative validation performance of networks with HyperNet-generated weights, we can effectively search over a wide range of architectures at the cost of a single training run. To facilitate this search, we develop a flexible mechanism based on memory read-writes that allows us to define a wide range of network connectivity patterns, with ResNet, DenseNet, and FractalNet blocks as special cases. We validate our method (SMASH) on CIFAR-10 and CIFAR-100, STL-10, ModelNet10, and Imagenet32x32, achieving competitive performance with similarly-sized hand-designed networks. Our code is available at this https URL
Subfields: Artificial Intelligence, Information Systems and Management, Neural Architecture Search & AutoML
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Yoshua Bengio
- Mohit Bansal
- Dacheng Tao
- Jianfeng Gao
- Sergey Levine
- Percy Liang
- Trevor Darrell
In Information Systems and Management
In Neural Architecture Search & AutoML
Citations in the corpus
- SNAS: Stochastic Neural Architecture Search (2018, background): “Ideas to design auxiliary mechanisms like performance prediction (Baker et al., 2017; Deng et al., 2017), iterative search (Liu et al., 2017a), hypernetwork generated weights (Brock et al., 2017) successfully accelerate NAS to certain degrees.”
- How to train your MAML (2018, background): “Recent work in meta-learning has produced state of the art results in a variety of settings (Wang et al., 2016; Ba et al., 2016; Zoph & Le, 2016; Andrychowicz et al., 2016; Vinyals et al., 2016; Li & Malik, 2016; Li et al., 2017; Antoniou et al 2017; Brock et al., 2017; Munkhdalai & Yu, 2017).”
- Weight Agnostic Neural Networks (2019, background): “The inner loop for training the weights of each candidate architecture before evaluation makes the search costly, although efforts have been made to improve efficiency [9, 65, 85].”
- Neural Architecture Optimization (2018, methodology): “Recently the design of neural network architectures has largely shifted from leveraging human knowledge to automatic methods, sometimes referred to as Neural Architecture Search (NAS) [42, 47, 48, 26, 36, 6, 39, 38, 27, 7, 6, 21].”
- A Comprehensive Survey of Neural Architecture Search (2020, methodology): “SMASH [23] proposes to train an auxiliary HyperNet [88] and use the auxiliary HyperNet to generate weights for other candidate network architectures.”
- BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models (2020, methodology): “Different from existing one-shot methods [2, 4, 23,26,36], our trained single-stage model offers a much wider coverage of model capacities, and more importantly, all child models are trained in a way such that they simultaneously reach excellent performance at the end of the search phase.”
- AutoEmb: Automated Embedding Dimensionality Search in Streaming Recommendations (2020, methodology): “For instance, ENAS[23] leverages a controller to sample subset of models, and SMASH[1] uses a hyper-network to generate weights for sampled networks.”
- DSNAS: Direct Neural Architecture Search Without Parameter Retraining (2020, methodology): “Different from all differentiable NAS methods, one-shot NAS only do architecture optimization in one-shot, before which they obtain a rough estimation of the graph through either pretraining [1, 7] or an auxiliary hypernetwork [2].”
- Fast Neural Network Adaptation via Parameter Remapping and Architecture Search (2020, methodology): “Recently, NAS methods based on one-shot model (Brock et al., 2017; Bender et al., 2018) or differentiable representations (Liu et al., 2019b; Cai et al., 2019; Fang et al., 2019b) achieve remarkable results with low search cost.”
- NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search (2020, neutral): “This can provide ground truth label for hypernetwork-based NAS methods (Zhang et al., 2019; Brock et al., 2018), which learn to generate parameters of an architecture.”
- DARTS+: Improved Differentiable Architecture Search with Early Stopping (2019, methodology): “Recently, one-shot methods [2, 3, 20, 25, 34] are proposed to reduce the search cost.”
- PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search (2019, methodology): “…generated networks (Cai et al., 2018), and later these methods were generalized into a more elegant framework named one-shot architecture search (Brock et al., 2018; Cai et al., 2019; Liu et al., 2019; Pham et al., 2018; Xie et al., 2019), in which an over-parameterized network or super-network…”
- FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search (2019, methodology): “As the conventional NAS approaches evaluate an enormous amount of models based on resourcedevouring training, recent attention is drawn to improve the estimation efficiency via parameter sharing [3, 11, 15, 27].”
- Evolving Robust Neural Architectures to Defend from Adversarial Attacks (2019, methodology): “Regarding the robustness of the architectures found, Table 2 shows that the final architecture found by DeepArchitect and SMASH were very susceptible to attacks, with error rate on adversarial samples of 75% and 82% respectively.”
- Searching for a Robust Neural Architecture in Four GPU Hours (2019, methodology): “Macro search algorithms aim to directly discover the entire neural networks [5, 4, 38, 46, 21].”
- BayesNAS: A Bayesian Approach for Neural Architecture Search (2019, background): “…subgraphs of a supergraph (the one-shot model) and shares weights between architectures that have edges of this super-graph in common (Saxena & Verbeek, 2016; Brock et al., 2018; Pham et al., 2018; Bender et al., 2018; Liu et al., 2019b; Cai et al., 2019; Xie et al., 2019; Zhang et al., 2019a;b).”
- A Survey on Neural Architecture Search (2019, methodology): “Hypernetworks Brock et al. (2018) propose the use of dynamic hypernetworks (Ha et al.”
- Single-Path NAS: Designing Hardware-Efficient ConvNets in less than 4 Hours (2019, background): “Gradient-based NAS in particular has gained increased popularity and has achieved state-of-the-art results [2].”