FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions

By Alvin Wan, Xiaoliang Dai, Peizhao Zhang, Zijian He, Yuandong Tian, Saining Xie, Bichen Wu, Matthew Yu, Tao Xu, Kan Chen and 2 more. 2020

334 citations recorded by the source corpus.

Differentiable Neural Architecture Search (DNAS) has demonstrated great success in designing state-of-the-art, efficient neural networks. However, DARTS-based DNAS's search space is small when compared to other search methods', since all candidate network layers must be explicitly instantiated in memory. To address this bottleneck, we propose a memory and computationally efficient DNAS variant: DMaskingNAS. This algorithm expands the search space by up to 10^14x over conventional DNAS, supporting searches over spatial and channel dimensions that are otherwise prohibitively expensive: input resolution and number of filters. We propose a masking mechanism for feature map reuse, so that memory and computational costs stay nearly constant as the search space expands. Furthermore, we employ effective shape propagation to maximize per-FLOP or per-parameter accuracy. The searched FBNetV2s yield state-of-the-art performance when compared with all previous architectures. With up to 421x less search cost, DMaskingNAS finds models with 0.9% higher accuracy, 15% fewer FLOPs than MobileNetV3-Small; and with similar accuracy but 20% fewer FLOPs than Efficient-B0. Furthermore, our FBNetV2 outperforms MobileNetV3 by 2.6% in accuracy, with equivalent model size. FBNetV2 models are open-sourced at https://github.com/facebookresearch/mobile-vision.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Neural Architecture Search & AutoML

Citations in the corpus

AutoHAS: Differentiable Hyper-parameter and Architecture Search (2020, methodology): “y choose RL instead of a differentiable strategy? Differentiable search methods have been extensively studied for its simplicity in many previous literature (Liu et al., 2019; Dong & Yang, 2019a; Wan et al., 2020; Xie et al., 2019), but these methods usually require much higher memory cost in or”
FBNetV3: Joint Architecture-Recipe Search using Neural Acquisition Function (2020, methodology): “Several works seek to address this issue, by training only subsets at a time [3] or by introducing approximations [36].”
ResNeSt: Split-Attention Networks (2020, methodology): “We follow the prior work of FBNetV2 [55], but use Split-Attention module instead of SE module in the MBConv block with the number of splits in [1,2,4].”
Auto-Agent-Distiller: Towards Efficient Deep Reinforcement Learning Agents via Neural Architecture Search (2020, methodology): “Later, DNAS [45, 80, 75, 8, 82] is proposed to update the weight and architecture in a differentiable manner through supernet weight sharing, which can reduce the search time to several hours [65].”
Pay Attention when Required (2020, background): “While recent works such as (Wu et al., 2019; Wan et al., 2020; Liu et al., 2019) explored using differential neural architecture search for designing ConvNets automatically to significantly improve accuracies and/or latencies, similar work for transformer models is limited.”
Sample-Efficient Neural Architecture Search by Learning Action Space (2019, neutral): “1 FBNetV2-F4( [42]) 238M 5.”
Retinex-inspired Unrolling with Cooperative Prior Architecture Search for Low-light Image Enhancement (2020, methodology): “Recently, differentiable search methods [16, 39, 32] formulated the super-network into a differentiable form with respect to a set of architectural parameters, so that they can be optimized by gradient descent and the search cost can be reduced to several hours.”
AutoHAS: Efficient Hyperparameter and Architecture Search. (2020, methodology): “Differentiable search methods have been extensively studied for its simplicity in many previous literature (Liu et al., 2019; Dong & Yang, 2019a; Wan et al., 2020; Xie et al., 2019), but these methods usually require much higher memory cost in order to train the entire super model.”
NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size (2020, methodology): “For example, how to design a FLOPs constrain loss to regularize the discovered architecture to be efficient [21], [43], [44]?”
BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search (2021, methodology): “Our BossNAS out performs recent NAS methods [66, 5] designed particularly for network size search spaces, proving the generalization ability of our method on specified search spaces and relatively small datasets.”
AMER: Automatic Behavior Modeling and Interaction Exploration in Recommender System (2020, background): “NAS has demonstrated great success in various tasks of computer vision [42, 65, 66, 69] and neural language processing [62, 71].”