Revisiting ResNets: Improved Training and Scaling Strategies

By Irwan Bello, William Fedus, Xianzhi Du, Ekin Dogus Cubuk, A. Srinivas, Tsung-Yi Lin, Jonathon Shlens, Barret Zoph. 2021

365 citations recorded by the source corpus.

Novel computer vision architectures monopolize the spotlight, but the impact of the model architecture is often conflated with simultaneous changes to training methodology and scaling strategies. Our work revisits the canonical ResNet (He et al., 2015) and studies these three aspects in an effort to disentangle them. Perhaps surprisingly, we find that training and scaling strategies may matter more than architectural changes, and further, that the resulting ResNets match recent state-of-the-art models. We show that the best performing scaling strategy depends on the training regime and offer two new scaling strategies: (1) scale model depth in regimes where overfitting can occur (width scaling is preferable otherwise); (2) increase image resolution more slowly than previously recommended (Tan&Le, 2019). Using improved training and scaling strategies, we design a family of ResNet architectures, ResNet-RS, which are 1.7x - 2.7x faster than EfficientNets on TPUs, while achieving similar accuracies on ImageNet. In a large-scale semi-supervised learning setup, ResNet-RS achieves 86.2% top-1 ImageNet accuracy, while being 4.7x faster than EfficientNet NoisyStudent. The training techniques improve transfer performance on a suite of downstream tasks (rivaling state-of-the-art self-supervised algorithms) and extend to video classification on Kinetics-400. We recommend practitioners use these simple revised ResNets as baselines for future research.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Optimization & Training

Citations in the corpus

Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers (2021, background): “While there have been many proposed scaling protocols for ConvNets (Tan and Le, 2019; Bello et al., 2021), there is still limited advice on scaling of transformer architectures, apart from (Kaplan et al.”
Combined Scaling for Zero-shot Transfer Learning (2021, methodology): “The second dimension is the model size, with representative works such as EfﬁcientNet, VITs and related works (Tan and Le, 2019, 2021; Tan et al., 2020; Dosovitskiy et al., 2021; Zhai et al., 2021; Bello et al., 2021).”
Scaling Vision Transformers (2021, background): “This strategy is revisited in [3, 47] to further improve the performance of ResNets [17].”
How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers (2021, background): “We also try two values for weight decay [23] which we found to work well, since increasing AugReg may need a decrease in weight decay [2].”
MLP-Mixer: An all-MLP Architecture for Vision (2021, background): “[4], who show that canonical ResNets are still near state-of-the-art, if one uses recent training and augmentation strategies.”
A ConvNet for the 2020s (2022, background): “Recent studies [7, 81] demonstrate that a set of modern training techniques can significantly enhance the performance of a simple ResNet-50 model.”
Bottleneck Transformers for Visual Recognition (2021, methodology): “BoT50 does not provide signiﬁcant gains over R50 on ImageNet though it does provide the bene-ﬁt of reducing the parameters while maintaining comparable computation (M.Adds).”
Simple Open-Vocabulary Object Detection with Vision Transformers (2022, background): “It is known that large Transformers trained on datasets of this size (such as ImageNet-1k) require carefully-tuned regularization and data augmentation to perform well [39,38,3].”
MaxViT: Multi-Axis Vision Transformer (2022, background): “More recent works have been focused on improving model and data efficiency, including sparse attention [1,21,64,86,96,99], improved locality [27,101], pyramidal designs [24,87, 97], improved training strategies [3,81,82,105], etc.”
Visual attention network (2022, background): “To further improve the usability, researchers put lots of effort in making the CNNs deeper [70,31,37,6,73], and lighter [33,67,104].”
Vision Models Are More Robust And Fair When Pretrained On Uncurated Images Without Supervision (2022, background): “More generally, a large body of work is dedicated to building efficient models with large capacity [9, 114, 118, 131].”
Contextual Transformer Networks for Visual Recognition (2021, methodology): “As shown in Table 6, compared to the SE-ResNetD-50 counterpart, our SE-CoTNetD-50 achieves better performances at a virtually negligible decrease in throughput.”
The Benchmark Lottery (2021, background): “…schemes, and also architectural extensions (e.g., Evolved Transformers [So et al., 2019], Synthesizers [Tay et al., 2020a], Universal Transformer [Dehghani et al., 2019], and Switch Transformers [Fedus et al., 2021]) as well as convolution-based models (e.g. lightweight and dynamic convolutions).”
Simple Training Strategies and Model Scaling for Object Detection (2021, positive): “[1] has demonstrated Squeezeand-Excitation module [19] and ResNet-D stem [34, 15] are both effective for classification model.”
ResMLP: Feedforward Networks for Image Classification With Data-Efficient Training (2021, result): “This is in line with recent work pointing out the importance of the training strategy over the model choice [3, 48].”
Knowledge distillation: A good teacher is patient and consistent (2021, methodology): “We investigate both directions: first, following [2], we train a ResNet-50 student”