Understanding and Simplifying One-Shot Architecture Search

By Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, Quoc V. Le. 2018

824 citations recorded by the source corpus.

Subfields: Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Neural Architecture Search & AutoML

Citations in the corpus

AutoGrow: Automatic Layer Growing in Deep Convolutional Networks (2019, neutral): “utogrow. time. For example, on CIFAR10 dataset, NAS used 22;400 Nvidia K40 GPU hours [39] and neural evolution [28] took 60;000 GPU hours [8] to perform the search. To accelerate NAS, one-shot models [29,26,2], DARTS [22] and NAS with Transferable Cell [40,20] were proposed. The search time reduces ”
DARTS: Differentiable Architecture Search (2018, background): “, 2018) and weight sharing/inheritance across multiple architectures (Elsken et al., 2017; Pham et al., 2018b; Cai et al., 2018; Bender et al., 2018), but the fundamental challenge of scalability remains.”
Neural Architecture Optimization (2018, background): “Different with NAO that tries to reduce the huge computational cost brought by the search algorithm, weight sharing aims to ease the huge complexity brought by massive child models via the one-shot model setup [5].”
Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution (2018, methodology): “One-Shot Bender et al. (2018).”
BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models (2020, methodology): “However, the absolute accuracies of predictions obtained from this supernetwork are typically much lower than those of models trained from scratch in stand-alone fashion [1].”
Real-Time Federated Evolutionary Neural Architecture Search (2020, neutral): “To ﬁx this issue, a sampling strategy is proposed by [39], [40], in which only one or two paths (a path is a sub-network connecting the inputs and outputs) are sampled from a complex neural network for training.”
Semi-Supervised Neural Architecture Search (2020, methodology): “To address this problem, one-shot NAS [2, 18, 15, 35] uses a supernet to include all candidate architectures via weight sharing and trains the supernet to reduce the training time.”
DSNAS: Direct Neural Architecture Search Without Parameter Retraining (2020, methodology): “Different from all differentiable NAS methods, one-shot NAS only do architecture optimization in one-shot, before which they obtain a rough estimation of the graph through either pretraining [1, 7] or an auxiliary hypernetwork [2].”
Fast Neural Network Adaptation via Parameter Remapping and Architecture Search (2020, methodology): “Recently, NAS methods based on one-shot model (Brock et al., 2017; Bender et al., 2018) or differentiable representations (Liu et al., 2019b; Cai et al., 2019; Fang et al., 2019b) achieve remarkable results with low search cost.”
Understanding and Robustifying Differentiable Architecture Search (2019, methodology): “…employing network morphisms [Cai et al., 2018a,b, Elsken et al., 2017, 2019b], weight sharing within one-shot models [Saxena and Verbeek, 2016, Bender et al., 2018, Pham et al., 2018] or multi-ﬁdelity optimization [Baker et al., 2017b, Falkner et al., 2018, Li et al., 2017, Zela et al.,…”
Resource Optimized Neural Architecture Search for 3D Medical Image Segmentation (2019, methodology): “ut patch size including all skip connections and operations. It is known that changes in batch normalization statistics hinder the training process early on when using drop-path regularization on NAS [1]. However, our training remains unaected because batch normalization is replaced with instance no”
FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search (2019, methodology): “To reduce the bias from different training orders, we don’t perform back-propagation and update parameters immediately for each model as in the previous works [2, 7].”
AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures (2019, neutral): “One-shot architecture search [2, 15] learning differentiable connections has also been studied.”
BayesNAS: A Bayesian Approach for Neural Architecture Search (2019, methodology): “Similar to other one-shot based NAS approaches (Bender et al., 2018; Zhang et al., 2019b; Liu et al., 2019b; Cai et al., 2019; Gordon et al., 2018), we also include (different or same) scaling scalars over all operations of all edges to control the information flow, denoted as woij which also…”
A Survey on Neural Architecture Search (2019, methodology): “This belief is supported by Bender et al. (2018) who show a correlation between the validation error obtained for a sample of the overparameterized network and the one obtained after training the corresponding architecture from scratch.”
Single-Path NAS: Designing Hardware-Efficient ConvNets in less than 4 Hours (2019, background): “As expected, naively branching out all paths is ineﬃcient due to an intrinsic limitation: the number of trainable parameters that need to be maintained and updated during the search grows linearly with respect to the number of candidate operations per layer [1].”
NAS-Unet: Neural Architecture Search for Medical Image Segmentation (2019, methodology): “SEARCH STRATEGY We first describe how to construct an over-parameterized network with all candidate path using recent works [20], [22], [50].”
DetNAS: Backbone Search for Object Detection (2019, methodology): “In one-shot NAS methods [6, 2, 1], the search space is encoded in a supernet which consists of all candidate architectures.”