EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
By Mingxing Tan, Quoc V. Le. 2019
25,169 citations recorded by the source corpus.
Convolutional Neural Networks (ConvNets) are commonly developed at a fixed resource budget, and then scaled up for better accuracy if more resources are available. In this paper, we systematically study model scaling and identify that carefully balancing network depth, width, and resolution can lead to better performance. Based on this observation, we propose a new scaling method that uniformly scales all dimensions of depth/width/resolution using a simple yet highly effective compound coefficient. We demonstrate the effectiveness of this method on scaling up MobileNets and ResNet. To go even further, we use neural architecture search to design a new baseline network and scale it up to obtain a family of models, called EfficientNets, which achieve much better accuracy and efficiency than previous ConvNets. In particular, our EfficientNet-B7 achieves state-of-the-art 84.4% top-1 / 97.1% top-5 accuracy on ImageNet, while being 8.4x smaller and 6.1x faster on inference than the best existing ConvNet. Our EfficientNets also transfer well and achieve state-of-the-art accuracy on CIFAR-100 (91.7%), Flowers (98.8%), and 3 other transfer learning datasets, with an order of magnitude fewer parameters. Source code is at this https URL.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Jianfeng Gao
- Mohit Bansal
- Yoshua Bengio
- Sergey Levine
- Percy Liang
- Dacheng Tao
- Luke S. Zettlemoyer
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Alan L. Yuille
In Efficiency & Compression
Citations in the corpus
- Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “As discussed in the paper, the first two models follow ResNet-50 and ResNet-101, and we use EfficientNetstyle (Tan & Le, 2019) scaling for the next three models which simultaneously scale the model width, the number of layers, and the input resolution to obtain models with roughly 4x, 16x, and 64x…”
- Experience Grounds Language (2020, background): “These advances have lead to parameter efficient models (Tan and Le, 2019) and real time object detection on embedded devices and phones (Rastegari et al.”
- Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, background): “…improves training efficiency by modifying loss functions (Clark et al., 2020), model structures/sparsities (Louizos et al., 2018; Gong et al., 2019; Tan & Le, 2019), backpropagation storage requirements (Gruslys et al., 2016), or learning rate schedules (Loshchilov & Hutter, 2017; Li et al.,…”
- A Constructive Prediction of the Generalization Error Across Scales (2019, methodology): “…dataset size, scaling up the model results in an initial decrease in test error, which then saturates to a level determined by the dataset size.1 This behavior has been noted by Tan & Le (2019) across varied model scaling methods, although they have not engaged with the dependency on dataset size.”
- One Epoch Is All You Need (2019, background): “Tan & Le (2019) showed that, for image classification, scaling up a model by jointly searching for the scaling factor of each scaling component (e.g. depth) with grid search leads to a dramatic improvement over scaling up one or two components only without extensive search.”
- Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers (2021, background): “Their paper demonstrates that the compound scaling rules (Tan and Le, 2019) derived in a small-scale regime, lead to Pareto-inefficient models when then extrapolated to larger scales.”
- Sustainable AI: Environmental Implications, Challenges and Opportunities (2021, background): “Beyond model accuracy and timing performance [45], [46], [47], [48], energy and carbon footprint can be directly incorporated into the cost function as optimization objectives to enable discovery of environmentallyfriendly models.”
- On Robustness and Transferability of Convolutional Neural Networks (2020, methodology): “& COTRAINING [71] IMAGENET RESNET 50 1 SIMCLR-1X-SELF-SUPERVISED SELF-SUPERVISED [6], FINE TUNING IMAGENET RESNET 50 1 SIMCLR-2X-SELF-SUPERVISED SELF-SUPERVISED [6], FINE TUNING IMAGENET RESNET 50 2 SIMCLR-4X-SELF-SUPERVISED SELF-SUPERVISED [6], FINE TUNING IMAGENET RESNET 50 4 SIMCLR-1X-FINE-TUNED-”
- Fixing the train-test resolution discrepancy: FixEfficientNet (2020, methodology): “We apply this method to the recent EfficientNet [6] architecture, which offers an excellent compromise between number of parameters and good performance.”
- Big Transfer (BiT): General Visual Representation Learning (2019, neutral): “7 [55] Pets 96.”
- Self-Training With Noisy Student Improves ImageNet Classification (2019, methodology): “First, we run an EfficientNet-B0 trained on ImageNet [82] over the JFT dataset [33, 15] to predict a label for each image.”
- Fixing the train-test resolution discrepancy (2019, methodology): “the art for a model learned from scratch is currently held by the EfficientNet-b7 [37] with 66M parameters using 600×600 images for training.”
- Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, methodology): “efficientnet-b0 [88].”
- On Empirical Comparisons of Optimizers for Deep Learning (2019, methodology): “Szegedy et al. (2016) used (cid:15) = 1 for RMSProp; Liu et al. (2019) reported that their results were sensitive to (cid:15) and set (cid:15) = 10 − 6 for Adam; Tan et al. (2019) and Tan and Le (2019) set (cid:15) = 10 − 3 for RMSProp, the latter achieving state-of-the-art ImageNet top-1 accuracy.”
- Early Convolutions Help Transformers See Better (2021, methodology): “For training and inference, ViTs use 224×224 resolution (we do not fine-tune at higher resolutions), while the CNNs use (often larger) optimized resolutions specified in [12, 39].”
- VOLO: Vision Outlooker for Visual Recognition (2021, background): “9 Efficientnet-b5 (IN-1 k) [24] 30 M 10.”