EfficientNetV2: Smaller Models and Faster Training
By Mingxing Tan, Quoc V. Le. 2021
4,600 citations recorded by the source corpus.
This paper introduces EfficientNetV2, a new family of convolutional networks that have faster training speed and better parameter efficiency than previous models. To develop this family of models, we use a combination of training-aware neural architecture search and scaling, to jointly optimize training speed and parameter efficiency. The models were searched from the search space enriched with new ops such as Fused-MBConv. Our experiments show that EfficientNetV2 models train much faster than state-of-the-art models while being up to 6.8x smaller. Our training can be further sped up by progressively increasing the image size during training, but it often causes a drop in accuracy. To compensate for this accuracy drop, we propose to adaptively adjust regularization (e.g., dropout and data augmentation) as well, such that we can achieve both fast training and good accuracy. With progressive learning, our EfficientNetV2 significantly outperforms previous models on ImageNet and CIFAR/Cars/Flowers datasets. By pretraining on the same ImageNet21k, our EfficientNetV2 achieves 87.3% top-1 accuracy on ImageNet ILSVRC2012, outperforming the recent ViT by 2.0% accuracy while training 5x-11x faster using the same computing resources. Code will be available at https://github.com/google/automl/tree/master/efficientnetv2.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Jianfeng Gao
- Mohit Bansal
- Dacheng Tao
- Yoshua Bengio
- Sergey Levine
- Trevor Darrell
- Jiashi Feng
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Jiashi Feng
- Jitendra Malik
- Chunhua Shen
In Efficiency & Compression
- Quoc V. Le
- Jiashi Feng
- Yunhe Wang
- Herv'e J'egou
- Xinchao Wang
- Qibin Hou
- Shuicheng Yan
- Mohammad Rastegari
Citations in the corpus
- Early Convolutions Help Transformers See Better (2021, methodology): “After pretraining, we fine-tune for 20 epochs on ImageNet-1k and use a small-scale grid search of lr while keeping wd at 0, similar to [13, 40].”
- VOLO: Vision Outlooker for Visual Recognition (2021, methodology): “Although the T2T-ViT can capture the local structural information and achieve superior performance on ImageNet than common CNN models, like ResNet and DenseNet, its performance still lags behind the advanced CNN models, such as EfficientNetV2 [70] and NFNet [8].”
- Combined Scaling for Zero-shot Transfer Learning (2021, methodology): “The second dimension is the model size, with representative works such as EfficientNet, VITs and related works (Tan and Le, 2019, 2021; Tan et al., 2020; Dosovitskiy et al., 2021; Zhai et al., 2021; Bello et al., 2021).”
- How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers (2021, background): “Ever since [18] first showed good results when pre-training BiT on ImageNet-21k, more architecture works have mentioned using it for select few experiments [10, 33, 32, 7], with [26] arguing more directly for the use of ImageNet-21k.”
- CoAtNet: Marrying Convolution and Attention for All Data Sizes (2021, methodology): “Recent works show that an improved inverted residual bottlenecks (MBConv [27, 35]), which is built upon depthwise convolutions, can achieve both high accuracy and better efficiency [5, 19].”
- A ConvNet for the 2020s (2022, neutral): “Table 1 (upper) shows the result comparison with two recent Transformer variants, DeiT [73] and Swin Transformers [45], as well as two ConvNets from architecture search - RegNets [54], EfficientNets [71] and EfficientNetsV2 [72].”
- CMT: Convolutional Neural Networks Meet Vision Transformers (2021, positive): “Comparison between CMT and EfficientNet-V2 [19].”
- Combined Scaling for Open-Vocabulary Image Classification (2022, methodology): “The second dimension is the model size, with representative works such as EfficientNet, VITs and related works (Tan and Le, 2019, 2021; Tan et al., 2020; Dosovitskiy et al., 2021; Zhai et al., 2021; Bello et al., 2021).”
- Inception Transformer (2022, neutral): “7 EfficientNetV2-L [73] 121 53 480 480 85.”
- FiT: Parameter Efficient Few-shot Transfer Learning for Personalized and Federated Image Classification (2022, methodology): “…mean accuracy over all 19 datasets ( Overall Acc ) and updateable parameter count ( Params ) for leading transfer learning methods using various backbones (backbone parameter count shown in parentheses) (Kolesnikov et al., 2019; Dosovitskiy et al., 2020; Tan & Le, 2021) pretrained on ImageNet-21k.”
- Solving ImageNet: a Unified Scheme for Training any Backbone to Top Results (2022, methodology): “3% lower accuracy than a tailor-made scheme [35].”
- Vision Transformers are Robust Learners (2021, methodology): “These models include EfficientNetV2 (Tan and Le 2021) with Global Context (GC) blocks (Cao et al.”
- CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers (2022, background): “Assuming the poses of all the agents are accurate and transmitted messages are synchronized, we propose a robust cooperative framework that can exploit the shared information across multiple agents to obtain a holistic BEV segmentation map.”
- A Generalist Framework for Panoptic Segmentation of Images and Videos (2022, methodology): “So on MS-COCO, we train the model with increasing resolutions, similar to [53, 54, 24].”