MnasNet: Platform-Aware Neural Architecture Search for Mobile
By Mingxing Tan, Ruoming Pang, Vijay Vasudevan, Quoc V. Le. 2018
3,454 citations recorded by the source corpus.
Designing convolutional neural networks (CNN) for mobile devices is challenging because mobile models need to be small and fast, yet still accurate. Although significant efforts have been dedicated to design and improve mobile CNNs on all dimensions, it is very difficult to manually balance these trade-offs when there are so many architectural possibilities to consider. In this paper, we propose an automated mobile neural architecture search (MNAS) approach, which explicitly incorporate model latency into the main objective so that the search can identify a model that achieves a good trade-off between accuracy and latency. Unlike previous work, where latency is considered via another, often inaccurate proxy (e.g., FLOPS), our approach directly measures real-world inference latency by executing the model on mobile phones. To further strike the right balance between flexibility and search space size, we propose a novel factorized hierarchical search space that encourages layer diversity throughout the network. Experimental results show that our approach consistently outperforms state-of-the-art mobile CNN models across multiple vision tasks. On the ImageNet classification task, our MnasNet achieves 75.2% top-1 accuracy with 78ms latency on a Pixel phone, which is 1.8× faster than MobileNetV2 with 0.5% higher accuracy and 2.3× faster than NASNet with 1.2% higher accuracy. Our MnasNet also achieves better mAP quality than MobileNets for COCO object detection. Code is at https://github.com/tensorflow/tpu/tree/master/models/official/mnasnet.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Yoshua Bengio
- Dacheng Tao
- Mohit Bansal
- Jianfeng Gao
- Trevor Darrell
- Percy Liang
- Sergey Levine
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Xiaogang Wang
- Trevor Darrell
- Alan L. Yuille
- Yi Yang
- Jitendra Malik
In Efficiency & Compression
Citations in the corpus
- EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks (2019, methodology): “Specifi-cally, we use the same search space as (Tan et al., 2019), and use ACC ( m ) × [ FLOPS ( m ) /T ] w as the optimization goal, where ACC ( m ) and FLOPS ( m ) denote the accuracy and FLOPS of model m , T is the target FLOPS and w =-0.07 is a hyperparameter for controlling the trade-off…”
- Visual Transformers: Token-based Image Representation and Processing for Computer Vision (2020, background): “Recently, people use neural architecture search [42, 10, 38, 9, 36, 35] to optimize network’s performance within a search space that consists of existing convolution operators.”
- Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, neutral): “[89] Tan, M.”
- On Empirical Comparisons of Optimizers for Deep Learning (2019, methodology): “Szegedy et al. (2016) used (cid:15) = 1 for RMSProp; Liu et al. (2019) reported that their results were sensitive to (cid:15) and set (cid:15) = 10 − 6 for Adam; Tan et al. (2019) and Tan and Le (2019) set (cid:15) = 10 − 3 for RMSProp, the latter achieving state-of-the-art ImageNet top-1 accuracy.”
- Primer: Searching for Efficient Transformers for Language Modeling (2021, background): “This approach, which we call an implicit efficiency objective by fixed training budget, contrasts previous architecture search works that explicitly aim to reduce training or inference step time when optimizing for efficiency [26, 27, 28, 29].”
- Squeeze-and-Excitation Networks (2017, neutral): “SE blocks can be used as atomic building blocks for these search algorithms, and were demonstrated to be highly effective in this capacity in concurrent work [45].”
- AutoML: A Survey of the State-of-the-Art (2019, methodology): “To do so, MnasNet [142] uses a customized weighted product to approximate a”
- Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution (2018, methodology): “Tan et al. (2018) use a weighted product method (Deb & Kalyanmoy, 2001) to obtain a scalarized objective.”
- Contrastive Representation Distillation (2019, neutral): “For MobileNetV2, ShuffleNetV1 and ShuffleNetV2, we use a learning rate of 0.01 as this learning rate is optimal for these models in a grid search, while 0.05 is optimal for other models.”
- AutoGAN-Distiller: Searching to Compress Generative Adversarial Networks (2020, neutral): “The idea of AutoML for model compression has been extended by a number of works, e.g, (Wu et al., 2019; Cheng et al., 2018; Tan & Le, 2019; Tan et al., 2019; Cheng et al., 2018), although most of them focused on compressing deep classifiers.”
- Benchmarking Deep Learning Models and Automated Model Design for COVID-19 Detection with Chest CT Scans (2020, methodology): “Many NAS studies [33], [34] use the MobileNetV2 structure to design the fac-”
- AutoHAS: Differentiable Hyper-parameter and Architecture Search (2020, methodology): “When searching for MBConv-based models [46, 41], we search for the kernel size from {3, 5, 7} and the expansion ratio from {3, 6}.”
- FBNetV3: Joint Architecture-Recipe Search using Neural Acquisition Function (2020, methodology): “Several of the most common techniques for NAS include reinforcement learning [50, 33], evolutionary algorithms [30], and DNAS [23, 38, 36, 8, 43].”
- A Comprehensive Survey of Neural Architecture Search (2020, background): “object detection [65, 75, 111, 118], semantic segmentation [63, 64, 120], adversarial learning [53], architectural scaling [114, 122, 124], multi-objective optimization [39, 115, 125], platform-aware [28, 34, 103, 117], data augmentation [121, 123] and so on. In addition, there is some work to consi”
- GreedyNAS: Towards Fast One-Shot NAS With Greedy Supernet (2020, methodology): “training. To train the obtained architecture, we use the same strategy as [1] for search space without SE. As for the augmented search space, we adopt a RMSProp optimizer with 0:9 momentum as Mnasnet [30]. Learning rate is increased from 0 to 0:064 in the first 5 epochs with batch size 512, and then ”
- BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models (2020, methodology): “We first plot the optimized and widely used exponentially decaying learning rate schedule for MobileNet-series [15, 16, 28], MNasNets [31] and EfficientNets [32] in Figure 2b.”
- Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, background): “The second trend is to design a tricky lightweight model to provide satisfying accuracy with fewer weights and parameters (Ma et al., 2018; Sandler et al., 2018; Tan et al., 2019).”