Fixing the train-test resolution discrepancy

By Hugo Touvron, Andrea Vedaldi, Matthijs Douze, Herv'e J'egou. 2019

484 citations recorded by the source corpus.

Data-augmentation is key to the training of neural networks for image classification. This paper first shows that existing augmentations induce a significant discrepancy between the typical size of the objects seen by the classifier at train and test time. We experimentally validate that, for a target test resolution, using a lower train resolution offers better classification at test time. We then propose a simple yet effective and efficient strategy to optimize the classifier performance when the train and test resolutions differ. It involves only a computationally cheap fine-tuning of the network at the test resolution. This enables training strong classifiers using small training images. For instance, we obtain 77.1% top-1 accuracy on ImageNet with a ResNet-50 trained on 128x128 images, and 79.8% with one trained on 224x224 image. In addition, if we use extra training data we get 82.5% with the ResNet-50 train with 224x224 images. Conversely, when training a ResNeXt-101 32x48d pre-trained in weakly-supervised fashion on 940 million public images at resolution 224x224 and further optimizing for test resolution 320x320, we obtain a test top-1 accuracy of 86.4% (top-5: 98.0%) (single-crop). To the best of our knowledge this is the highest ImageNet single-crop, top-1 and top-5 accuracy to date.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, neutral): “It is often beneﬁcial to ﬁne-tune at higher resolution than pre-training (Touvron et al., 2019; Kolesnikov et al., 2020).”
Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “For the ViT-L/14 we also pre-train at a higher 336 pixel resolution for one additional epoch to boost performance similar to FixRes (Touvron et al., 2019).”
On Robustness and Transferability of Convolutional Neural Networks (2020, methodology): “Increasing the resolution at which images are tested leads to an improvement in accuracy across different architectures [63, 64].”
Fixing the train-test resolution discrepancy: FixEfficientNet (2020, methodology): “Touvron et al. show [7] that this separate optimization has a detrimental effect on the test-time performance of models.”
Big Transfer (BiT): General Visual Representation Learning (2019, background): “ixel space. In these cases we omit ipping or cropping when appropriate. Recent work has shown that existing augmentation methods introduce inconsistency between training and test resolutions for CNNs [57]. Therefore, it is common to scale up the resolution by a small factor at test time. As an alter”
Self-Training With Noisy Student Improves ImageNet Classification (2019, methodology): “Similar to [71], we fix the shallow layers during finetuning.”
Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, methodology): “The top-right green cluster on both plots consists of several ResNet and ResNeXt models trained on 1 billion Instagram images [56, 92, 104] and EfficientNet-L2 (NoisyStudent) trained on the JFT-300M dataset of 300 million images [102].”
VOLO: Vision Outlooker for Visual Recognition (2021, methodology): “Later stateof-the-art models [30, 53, 63] mostly utilize different training or optimization methods or finetuning techniques to improve EfficientNet.”
Training data-efficient image transformers & distillation through attention (2020, background): “[47] show that it is desirable to use a lower training resolution and fine-tune the network at the larger resolution.”
Grafit: Learning fine-grained image representations with coarse labels (2020, methodology): “Forﬁne-tuning results in Table 8 we additionally use Cutmix [68] and FixRes [53] during ﬁne-tuning and we ﬁne-tune with more epochs (1000 for Flowers [41] and Cars [35], 300 for Food-101 [7] and iNatu-ralist [30, 31]).”
Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “[113] Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou.”
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, methodology): “The same pattern holds (with additional performance gains) as model size increases: Reserve-L improves over prior work by 7.6%.”
Combined Scaling for Zero-shot Transfer Learning (2021, neutral): “While we can increase this resolution to gain performance (Tan and Le, 2019, 2021; Touvron et al., 2019; Radford et al., 2021; Jia et al., 2021), we choose not to do this and instead, reserve our computational resources for scaling up our model and our batch size.”
High-Performance Large-Scale Image Recognition Without Normalization (2021, methodology): “Following Touvron et al. (2019), we evaluate images at inference at a slightly higher resolution than we train at, chosen for each variant as approximately 33% larger than the train resolution.”
How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers (2021, neutral): “Fine-tuning was done both at the original resolution (224), as well as at a higher resolution (384) as described in [35].”