ResNet strikes back: An improved training procedure in timm

By Ross Wightman, Hugo Touvron, Herv'e J'egou. 2021

625 citations recorded by the source corpus.

The influential Residual Networks designed by He et al. remain the gold-standard architecture in numerous scientific publications. They typically serve as the default architecture in studies, or as baselines when new architectures are proposed. Yet there has been significant progress on best practices for training neural networks since the inception of the ResNet architecture in 2015. Novel optimization&data-augmentation have increased the effectiveness of the training recipes. In this paper, we re-evaluate the performance of the vanilla ResNet-50 when trained with a procedure that integrates such advances. We share competitive training settings and pre-trained models in the timm open-source library, with the hope that they will serve as better baselines for future work. For instance, with our more demanding training setting, a vanilla ResNet-50 reaches 80.4% top-1 accuracy at resolution 224x224 on ImageNet-val without extra data or distillation. We also report the performance achieved with popular models with our training procedure.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Optimization & Training

Citations in the corpus

Scaling Vision Transformers (2021, background): “This strategy is revisited in [3, 47] to further improve the performance of ResNets [17].”
Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement (2023, methodology): “We follow [42, 64] and use state-of-the-art recipes, including optimizers, hyperpa-rameters, and learning schedules, specific to each model on the ImageNet.”
The Efficiency Misnomer (2021, background): “That said, it is still common practice to parameter-match models to make ‘fair’ comparisons (Mehta et al., 2020; Lee-Thorp et al., 2021; Tay et al., 2020a; Xue et al., 2021; Wightman et al., 2021), even if one model is in reality slower or faster than another.”
A ConvNet for the 2020s (2022, background): “Recent studies [7, 81] demonstrate that a set of modern training techniques can significantly enhance the performance of a simple ResNet-50 model.”
CVNets: High Performance Library for Computer Vision (2022, methodology): “…strategies are visualized in Figure 2a and discussed below: • Single-scale with fixed batch size (SSc-FBS): This method is the default sampling strategy in most deep learning frameworks (e.g., PyTorch, Tensorflow, and MixNet) and libraries built on top of them (e.g., the timm library [21]).”
Inception Transformer (2022, neutral): “(∼ 50 M ) CNN RSB-ResNet-101 [47, 63] 45 7.”
RTMDet: An Empirical Study of Designing Real-Time Object Detectors (2022, methodology): “Finally, applying a pre-trained ImageNet backbone through the RSB [78] training strategy leads to a further 0.3% increase in AP.”
Sigmoid Loss for Language Image Pre-Training (2023, positive): “In supervised classiﬁcation, the sigmoid loss has already been shown to be slightly more effective and robust than the softmax loss [3, 47].”
Focal Modulation Networks (2022, methodology): “…are integrated to SA and bring significant gains [74, 22, 39, 20] or the vice versa [64]; ( ii ) ResNets have closed the gap to ViTs using similar data augmentation and regularization strategies [73], and replacing SA with (dynamic) depth-wise convolution [25, 47] can also slightly surpass Swin.”
PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategies (2022, positive): “Training strategies are studied recently in [2, 45, 24] on image classiﬁcation.”
InceptionNeXt: When Inception Meets ConvNeXt (2023, methodology): “With advanced training techniques in DeiT [64] and Swin [39], the work of “ResNet strikes back” [72] shows that the performance of ResNet-50 can rise by 2.3%, up to 78.4%.”
DiffusionDet: Diffusion Model for Object Detection (2022, methodology): “Then, we evaluate each model instance with ten different random seeds to measure the distribution of performance, inspired by [69, 96].”