Searching for Fast Model Families on Datacenter Accelerators
By Sheng Li, Mingxing Tan, Ruoming Pang, Andrew Li, Liqun Cheng, Quoc V. Le, N. Jouppi. 2021
40 citations recorded by the source corpus.
Neural Architecture Search (NAS), together with model scaling, has shown remarkable progress in designing high accuracy and fast convolutional architecture families. However, as neither NAS nor model scaling considers sufficient hardware architecture details, they do not take full advantage of the emerging datacenter (DC) accelerators. In this paper, we search for fast and accurate CNN model families for efficient inference on DC accelerators. We first analyze DC accelerators and find that existing CNNs suffer from insufficient operational intensity, parallelism, and execution efficiency and exhibit FLOPs-latency nonproportionality. These insights let us create a DC-accelerator-optimized search space, with space-to-depth, space-to-batch, hybrid fused convolution structures with vanilla and depthwise convolutions, and block-wise activation functions. We further propose a latency-aware compound scaling (LACS), the first multi-objective compound scaling method optimizing both accuracy and latency. Our LACS discovers that network depth should grow much faster than image size and network width, which is quite different from the observations from previous compound scaling. With the new search space and LACS, our search and scaling on datacenter accelerators results in a new model series named EfficientNet-X. EfficientNet-X is up to more than 2X faster than Efficient-Net (a model series with state-of-the-art trade-off on FLOPs and accuracy) on TPUv3 and GPUv100, with comparable accuracy. EfficientNet-X is also up to 7X faster than recent RegNet and ResNeSt on TPUv3 and GPUv100. Source code is at https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet/tpu
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Joseph E. Gonzalez
- Chen Liang
- David R. So
- Jianfeng Gao
- Mohit Bansal
- Richard Socher
- Joseph Dean
In Computer Vision and Pattern Recognition
- Quoc V. Le
- Joseph E. Gonzalez
- Chen Liang
- David R. So
- Luc Van Gool
- Trevor Darrell
- Dacheng Tao
- Joseph Dean
In Electrical and Electronic Engineering
- Chen Liang
- David A. Patterson
- N. Jouppi
- Brian Towles
- Lifeng Nai
- Quoc V. Le
- David R. So
- Joseph E. Gonzalez
Citations in the corpus
- The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink (2022, methodology): “NAS has also been applied to find models that have better quality and run faster by adapting them to a given processor [15].”
- TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings (2023, methodology): “To enable Pareto-optimizations over quality and performance for DNN models, we developed platform-aware neural architecture search (PA-NAS) at scale to tailor DNN models for TPU v4 supercomputers automatically [32].”