Large Scale Learning of General Visual Representations for Transfer

By Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, Neil Houlsby. 2019

104 citations recorded by the source corpus.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “While Mahajan et al. (2018) and Kolesnikov et al. (2019) trained their models for accelerator years on millions to billions of images, VirTex, ICMLM, and ConVIRT trained for accelerator days on one to two hundred thousand images.”
Self-Training With Noisy Student Improves ImageNet Classification (2019, background): “Big Transfer (BiT-L) [43]† 928M 300M weakly labeled images from JFT 87.”
The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization (2020, neutral): “We also tune an ImageNet21K model [19].”
Big Self-Supervised Models are Strong Semi-Supervised Learners (2020, background): “The effectiveness of big models have been demonstrated on supervised learning [64–67], fine-tuning supervised models on a few examples [68], and unsupervised learning on language [9, 69, 10, 70].”
What is being transferred in transfer learning? (2020, background): “[24] propose a heuristic for setting the hyperparameters for transfer.”
Supervised Contrastive Learning (2020, background): “However, in practice, most proposed alternatives have not worked better for large-scale datasets, such as ImageNet [7], as evidenced by the continued use of cross-entropy to achieve state of the art results [5, 6, 55, 25].”
MoPro: Webly Supervised Learning with Momentum Prototypes (2020, background): “Several works have exploited the scalable source of web images and demonstrated promising results with weblysupervised representation learning (Mahajan et al., 2018; Sun et al., 2017; Li et al., 2017; Kolesnikov et al., 2020).”
Exploring Visual Engagement Signals for Representation Learning (2021, background): “This motivates a plethora of work investigating weakly-supervised [73, 55, 41, 42, 47], semisupervised [86, 87, 84] and self-supervised learning [3, 96, 14, 4, 22, 48, 90].”
Consistent Estimators for Learning to Defer to an Expert (2020, methodology): “94 to simulate SOTA performance on CIFAR-100 with 93.8% test accuracy [KBZ + 19], otherwise the expert predicts uniformly at random.”
How Useful Is Self-Supervised Pretraining for Visual Tasks? (2020, background): “Similar findings are shown when pretraining with noisy labels [23].”
Characterizing signal propagation to close the performance gap in unnormalized ResNets (2021, result): “As one exception, we note that the combination of GroupNorm with Weight Standardization (Qiao et al., 2019) was recently identiﬁed as a promising alternative to BatchNorm in ResNet-50 (Kolesnikov et al., 2019).”