Big Transfer (BiT): General Visual Representation Learning

By Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, Neil Houlsby. 2019

1,392 citations recorded by the source corpus.

Transfer of pre-trained representations improves sample efficiency and simplifies hyperparameter tuning when training deep neural networks for vision. We revisit the paradigm of pre-training on large supervised datasets and fine-tuning the model on a target task. We scale up pre-training, and propose a simple recipe that we call Big Transfer (BiT). By combining a few carefully selected components, and transferring using a simple heuristic, we achieve strong performance on over 20 datasets. BiT performs well across a surprisingly wide range of data regimes -- from 1 example per class to 1M total examples. BiT achieves 87.5% top-1 accuracy on ILSVRC-2012, 99.4% on CIFAR-10, and 76.3% on the 19 task Visual Task Adaptation Benchmark (VTAB). On small datasets, BiT attains 76.8% on ILSVRC-2012 with 10 examples per class, and 97.0% on CIFAR-10 with 10 examples per class. We conduct detailed analysis of the main components that lead to high transfer performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, positive): “We de-duplicate the pre-training datasets w.r.t. the test sets of the downstream tasks following Kolesnikov et al. (2020).”
On Robustness and Transferability of Convolutional Neural Networks (2020, methodology): “FixRes arguably does not lead to improvements as it was already applied in BiT as a part of the BiT HyperRule.”
Are we done with ImageNet? (2020, methodology): “We start by assembling a set of 19 models [3, 27, 4, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37] which vary along a number of dimensions, including architecture, training objective and usage of external data (see appendix A).”
Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, methodology): “We examine two pipeline specifications: one that trains a ResNet-50 model (He et al., 2016) on ImageNet, and another that pretrains a ResNet-101x3 Big Transfer (BiT) model (Kolesnikov et al., 2019) on the JFT-300M dataset (Sun et al., 2017) then fine-tunes it on ImageNet.”
Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, neutral): “…includes models trained on (i) Facebook’s collection of 1 billion Instagram images [56, 104], (ii) the YFCC 100 million dataset [104], (iii) Google’s JFT 300 million dataset [82, 102], (iv) a subset of OpenImages [98], or (v) a subset of the full ImageNet dataset of 21,841 classes [11, 49, 99].”
Early Convolutions Help Transformers See Better (2021, methodology): “[13] also introduced a “hybrid ViT” architecture that blends a modified ResNet [19] (BiT-ResNet [22]) with a transformer encoder.”
An Empirical Study of Training Self-Supervised Vision Transformers (2021, methodology): “After the first version of this manuscript, Lucas Beyer, an author of BiT [24] and ViT [15], echoed that “the exact same behaviour” was observed for supervised BiT-ResNet in ImageNet-21k.”
Self-supervised Pretraining of Visual Features in the Wild (2021, background): “Beneﬁting from the advances in distributed training [18], several works have shown the advantages of pretraining on large curated image datasets with weak-supervised learning [27, 35], semi-supervised learning [54] or supervised training on hundreds of millions of ﬁltered images [29, 47].”
Projected GANs Converge Faster (2021, neutral): “Using pretrained representations has become ubiquitous in computer vision [29, 30, 48]”
Sharpness-Aware Minimization for Efficiently Improving Generalization (2020, background): “, 2018), BiT-M/L (Kolesnikov et al., 2020), KDforAA (Wei et al.”
Robust Speech Recognition via Large-Scale Weak Supervision (2022, background): “, 2015) to much larger but weakly supervised datasets significantly improves the robustness and generalization of models (Mahajan et al., 2018; Kolesnikov et al., 2020).”
SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network (2021, neutral): “Transfer learning is a popular technique found in computer vision [54, 19, 22] and natural language processing [14, 26].”
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, methodology): “In computer vision and natural language processing, the best performing models are often pre-trained on a large dataset before being finetuned on data from the target task (Donahue et al., 2014; Yosinski et al., 2014; Sharif Razavian et al., 2014; Girshick et al., 2014; Mahajan et al., 2018; Kornbli”
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, background): “Recent prior work on static images that demonstrates empirical improvements by increasing dataset size – all the way up to JFT-3B [69, 34, 91, 128].”
Florence: A New Foundation Model for Computer Vision (2021, methodology): “Our model outperforms BiT (Kolesnikov et al., 2020) with larger model size and ALIGN (Jia et al., 2021) trained from more data in terms of Top-1 and Top-5 accuracy.”