ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware

By Han Cai, Ligeng Zhu, Song Han. 2018

2,104 citations recorded by the source corpus.

Neural architecture search (NAS) has a great impact by automatically designing effective neural network architectures. However, the prohibitive computational demand of conventional NAS algorithms (e.g. $10^4$ GPU hours) makes it difficult to \emph{directly} search the architectures on large-scale tasks (e.g. ImageNet). Differentiable NAS can reduce the cost of GPU hours via a continuous representation of network architecture but suffers from the high GPU memory consumption issue (grow linearly w.r.t. candidate set size). As a result, they need to utilize~\emph{proxy} tasks, such as training on a smaller dataset, or learning with only a few blocks, or training just for a few epochs. These architectures optimized on proxy tasks are not guaranteed to be optimal on the target task. In this paper, we present \emph{ProxylessNAS} that can \emph{directly} learn the architectures for large-scale target tasks and target hardware platforms. We address the high memory consumption issue of differentiable NAS and reduce the computational cost (GPU hours and GPU memory) to the same level of regular training while still allowing a large candidate set. Experiments on CIFAR-10 and ImageNet demonstrate the effectiveness of directness and specialization. On CIFAR-10, our model achieves 2.08\% test error with only 5.7M parameters, better than the previous state-of-the-art architecture AmoebaNet-B, while using 6$\times$ fewer parameters. On ImageNet, our model achieves 3.1\% better top-1 accuracy than MobileNetV2, while being 1.2$\times$ faster with measured GPU latency. We also apply ProxylessNAS to specialize neural architectures for hardware with direct hardware metrics (e.g. latency) and provide insights for efficient CNN architecture design.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks (2019, methodology): “Unlike (Tan et al., 2019; Cai et al., 2019), here we optimize FLOPS rather than latency since we are not targeting any speciﬁc hardware device.”
Primer: Searching for Efficient Transformers for Language Modeling (2021, background): “This approach, which we call an implicit efficiency objective by fixed training budget, contrasts previous architecture search works that explicitly aim to reduce training or inference step time when optimizing for efficiency [26, 27, 28, 29].”
AutoML: A Survey of the State-of-the-Art (2019, methodology): “For saving memory consumption, In [114], a new path-level pruning method is introduced.”
MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulation (2020, methodology): “Moreover, recent proposed NAS methods achieve a higher accuracy than our method [20, 11, 2].”
AutoGAN-Distiller: Searching to Compress Generative Adversarial Networks (2020, positive): “We apply Gumbel-Softmax (Gumbel, 1948) to approximate differentiable sampling for ϕ based on γ as (Cai et al., 2018).”
AutoHAS: Differentiable Hyper-parameter and Architecture Search (2020, methodology): “The model size and latency of the searched architectures have been reduced by Pareto optimization [46, 50, 6, 5] and enlarged search space of network size [5, 10].”
FBNetV3: Joint Architecture-Recipe Search using Neural Acquisition Function (2020, background): “Conv 3 (16, 24, 2) 1 2 hswish MBConv [3, 5] 1 (16, 24, 2) (1, 4) 1 N hswish MBConv [3, 5] (4, 7) / (2, 5) (20, 32, 4) (4, 7) 2 N hswish MBConv [3, 5] (4, 7) / (2, 5) (24, 48, 4) (4, 7) 2 Y hswish MBConv [3, 5] (4, 7)(1) / (2, 5)(2) (56, 84, 4) (4, 8) 2 N hswish MBConv [3, 5] (4, 7)(1) / (2, 5)(2) (9”
A Comprehensive Survey of Neural Architecture Search (2020, methodology): “In NAS work based on gradient optimization [17, 102, 103], the joint optimization of supernet weights and architectural parameters will also introduce bias between sub-models.”
Are Labels Necessary for Neural Architecture Search? (2020, background): “This suggests it might be desirable to perform architecture search on the target dataset directly, as also observed in other work [3].”
GreedyNAS: Towards Fast One-Shot NAS With Greedy Supernet (2020, methodology): “Following [1, 4], we adopt the same macro-structure of supernet for fair comparison as shown in Table 5 (see supplementary materials).”
BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models (2020, background): “For example, one proposed solution is to retrain a separate model for each device of interest and each latency budget of interest [5, 33].”
Semi-Supervised Neural Architecture Search (2020, methodology): “Specifically, it significantly outperforms the baseline algorithm NAO based on which SemiNAS is mainly implemented by 1.0%, and outperforms ProxylessNAS where our search space is based on by 0.5%.”
DSNAS: Direct Neural Architecture Search Without Parameter Retraining (2020, methodology): “Therefore, NAS is always formulated as a hyper-parameter optimization problem, whose algorithmic realization spans evolution algorithm [21, 7], reinforcement learning [27], Bayesian optimization [9], Monte Carlo Tree Search [24], and differentiable architecture search [14, 26, 3].”
On Robustness of Neural Architecture Search Under Label Noise (2020, methodology): “…optimization (Kandasamy et al., 2018; Jin et al., 2019), reinforcement learning (Zoph and Le, 2017; Cai et al., 2018; Pham et al., 2018; Zoph et al., 2018), evolutionary algorithms (Real et al., 2017, 2019) and gradient-based optimization (Luo et al., 2018; Cai et al., 2019; Liu et al., 2019).”
Fast Neural Network Adaptation via Parameter Remapping and Architecture Search (2020, methodology): “As most differentiable NAS methods (Liu et al., 2019b; Cai et al., 2019; Wu et al., 2018) do, we relax every layer as a weighted sum of all candidate operations. where O denotes the operation set, α io denotes the architecture parameter of operation o in the i th layer, and x denotes the input…”
AtomNAS: Fine-Grained End-to-End Neural Architecture Search (2019, background): “Cai et al. (2019) and Wu et al. (2019) utilize supernets with multiple choices in each layer to accommodate a sampled network on the GPU.”