Single-Path NAS: Designing Hardware-Efficient ConvNets in less than 4 Hours

By Dimitrios Stamoulis, Ruizhou Ding, Di Wang, Dimitrios Lymberopoulos, Bodhi Priyantha, Diana Marculescu. 2019

291 citations recorded by the source corpus.

Can we automatically design a Convolutional Network (ConvNet) with the highest image classification accuracy under the runtime constraint of a mobile device? Neural architecture search (NAS) has revolutionized the design of hardware-efficient ConvNets by automating this process. However, the NAS problem remains challenging due to the combinatorially large design space, causing a significant searching time (at least 200 GPU-hours). To alleviate this complexity, we propose Single-Path NAS, a novel differentiable NAS method for designing hardware-efficient ConvNets in less than 4 hours. Our contributions are as follows: 1. Single-path search space: Compared to previous differentiable NAS methods, Single-Path NAS uses one single-path over-parameterized ConvNet to encode all architectural decisions with shared convolutional kernel parameters, hence drastically decreasing the number of trainable parameters and the search cost down to few epochs. 2. Hardware-efficient ImageNet classification: Single-Path NAS achieves 74.96% top-1 accuracy on ImageNet with 79ms latency on a Pixel 1 phone, which is state-of-the-art accuracy compared to NAS methods with similar constraints (<80ms). 3. NAS efficiency: Single-Path NAS search cost is only 8 epochs (30 TPU-hours), which is up to 5,000x faster compared to prior work. 4. Reproducibility: Unlike all recent mobile-efficient NAS methods which only release pretrained models, we open-source our entire codebase at: this https URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Efficiency & Compression

Citations in the corpus

Sustainable AI: Environmental Implications, Challenges and Opportunities (2021, background): “Furthermore, when training is decoupled from NAS, sub-networks tailoring to specialized system hardware can be selected without additional training [51], [52], [53], [54].”
AutoML: A Survey of the State-of-the-Art (2019, methodology): “Network morphism based algorithms [18, 19] can also inherit the weights of previous architectures, and single-path NAS [132] uses a single-path over-parameterized ConvNet to encode all architectural decisions with shared convolutional”
GreedyNAS: Towards Fast One-Shot NAS With Greedy Supernet (2020, methodology): “ProxylessNAS [1] alleviates this issue by factorizing the searching into multiple binary selection tasks while Single-Path-NAS [29] uses superkernels to encode all operation choices.”
BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models (2020, methodology): “[1,3,9,22,25,30,35]) or otherwise post-process them (e.”
Semi-Supervised Neural Architecture Search (2020, background): “[18, 17, 15, 5, 36, 4, 27, 9] also leverage the one-shot idea to perform efficient search while using different search algorithms.”
AtomNAS: Fine-Grained End-to-End Neural Architecture Search (2019, methodology): “Stamoulis et al. (2019a) propose to use different kernel sizes in each layer of the supernet and reuse the weights of larger kernels for small kernels.”
DARTS+: Improved Differentiable Architecture Search with Early Stopping (2019, methodology): “…it should be noticed that many recent progresses of DARTS could use “early stop-ping” to achieve better results, and the proposed “early stop-ping” criteria could be applied to many other types of search spaces, including the RandWire search space [35], mobile convnets search space [30], etc.”
FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search (2019, methodology): “Many one-shot variants emerge and the design and training strategies of supernet can be roughly classified into three categories: training the whole supernet based on dropconnect tricks [2], jointly training the weights of choices and network parameters (in turns) [4, 11, 27], and training it in a s”
Single Path One-Shot Neural Architecture Search with Uniform Sampling (2019, background): “We note that sampling a path according to architecture distribution during optimization is already used in previous weight sharing approaches [22, 4, 31, 28, 6, 20].”
Auto-Agent-Distiller: Towards Efficient Deep Reinforcement Learning Agents via Neural Architecture Search (2020, methodology): “Later, DNAS [45, 80, 75, 8, 82] is proposed to update the weight and architecture in a differentiable manner through supernet weight sharing, which can reduce the search time to several hours [65].”
FasterSeg: Searching for Faster Real-time Semantic Segmentation (2019, methodology): “Inspired by (Yu et al., 2018c) and (Stamoulis et al., 2019), from slim to wide our connections incrementally take larger subsets of input/output dimensions from the superkernel.”
Mitigating Forgetting in Online Continual Learning via Instance-Aware Parameterization (2020, methodology): “To reduce the size of the meta-graph, we adopt the concept from [28].”
SCARLET-NAS: Bridging the Gap between Stability and Scalability in Weight-sharing Neural Architecture Search (2019, unknown): “1 SingPath NAS [26] 365 75.”
NetAdaptV2: Efficient Neural Architecture Search with Fast Super-Network Training and Architecture Optimization (2021, methodology): “3), and superkernel [12, 21, 22] to efficiently train the super-network that contains sub-networks with different layer widths, network depths, and kernel sizes.”
Can Weight Sharing Outperform Random Architecture Search? An Investigation With TuNAS (2020, methodology): “Recently, Neural Architecture Search has been used intensively to find architectures that have better tradeoff between accuracy and latency [38, 41, 8, 5, 37], FLOPS [39], power consumption [16], and memory usage [10].”
Evolving Normalization-Activation Layers (2020, background): “, [20,25–50], but has a very different goal.”
Dynamic Routing Networks (2019, background): “For a small 10 cells DRNet, with 8 connections per cell and 5 candidate operations per connection, there are (2(5) − 1)8·10 ≈ 2 · 10(119) possible candidate architectures of different branch selections, which is orders of magnitudes larger than the search space of conventional NAS [25, 20, 39, 4, 31”