Fixing the train-test resolution discrepancy: FixEfficientNet

By Hugo Touvron, Andrea Vedaldi, Matthijs Douze, Herv'e J'egou. 2020

113 citations recorded by the source corpus.

This note complements the paper "Fixing the train-test resolution discrepancy" that introduced the FixRes method. First, we show that this strategy is advantageously combined with recent training recipes from the literature. Most importantly, we provide new results for the EfficientNet architecture. The resulting network, called FixEfficientNet, significantly outperforms the initial architecture with the same number of parameters. For instance, our FixEfficientNet-B0 trained without additional training data achieves 79.3% top-1 accuracy on ImageNet with 5.3M parameters. This is a +0.5% absolute improvement over the Noisy student EfficientNet-B0 trained with 300M unlabeled images and +1.7% compared to the EfficientNet-B0 trained with adversarial examples. An EfficientNet-L2 pre-trained with weak supervision on 300M unlabeled images and further optimized with FixRes achieves 88.5% top-1 accuracy (top-5: 98.7%), which establishes the new state of the art for ImageNet with a single crop.This improvement in performance transfers to the experimental setting of ImageNet-v2, that is less prone to overfitting, for which we establish the new state of the art.

Subfields: Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

On Robustness and Transferability of Convolutional Neural Networks (2020, background): “Increasing the resolution at which images are tested leads to an improvement in accuracy across different architectures [63, 64].”
Fixing the train-test resolution discrepancy (2019, background): “See our note [39] for results and details.”
Self-supervised Pretraining of Visual Features in the Wild (2021, background): “of literature is dedicated to building scale efficient models with large capacity [48, 53, 49].”
Training data-efficient image transformers & distillation through attention (2020, methodology): “Convolutional neural networks have optimized, both in terms of architecture and optimization during almost a decade, including through extensive architecture search that is prone to overfiting, as it is the case for instance for EfficientNets [48].”
Grafit: Learning fine-grained image representations with coarse labels (2020, background): “Image classification now achieves a performance that meets many application needs [27, 37, 54].”
ORBIT: A Real-World Few-Shot Dataset for Teachable Object Recognition (2021, background): “Object recognition systems have made spectacular advances in recent years [42, 47, 43, 37, 14, 30, 36] however, most systems still rely on training datasets with 100s to 1,000s of high-quality, labeled examples per object category.”
Going deeper with Image Transformers (2021, neutral): “[64], EfficientNet [14, 63, 72], Fix-EfficientNet [66] and NFNets [8].”
Meta Pseudo Labels (2020, methodology): “EfficientNet-B7 + FixRes [70] 66M − 85.”
EfficientNetV2: Smaller Models and Faster Training (2021, background): “Following (Touvron et al., 2020), our maximum image size for training is about 20% smaller than inference, but we don’t finetune any layers after training.”
Review of Image Classification Algorithms Based on Convolutional Neural Networks (2021, background): “[182,261] proposes a training strategy that employs different train and test resolutions, to optimize the classifier performance.”
DeiT III: Revenge of the ViT (2022, unknown): “The comparison between Imagenet-val and -v2 is a way to quantify overfitting [54], or at least the better capability to generalize in a nearby setting without any fine-tuning1.”
SqueezeBERT: What can computer vision teach NLP about efficient neural networks? (2020, methodology): “On the ImageNet [28] image clas-siﬁcation task, from the year 2016 (ResNet-101 [23]) to 2020 (FixEfﬁcientNet-D0 [88]), there is a 20x reduction in the number of ﬂoating-point operations (FLOPs) required, while accuracy has actually improved.”