A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark

By Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, P. Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, André Susano Pinto, Maxim Neumann, Alexey Dosovitskiy and 7 more. 2019

558 citations recorded by the source corpus.

Representation learning promises to unlock deep learning for the long tail of vision tasks without expensive labelled datasets. Yet, the absence of a unified evaluation for general visual representations hinders progress. Popular protocols are often too constrained (linear classification), limited in diversity (ImageNet, CIFAR, Pascal-VOC), or only weakly related to representation quality (ELBO, reconstruction error). We present the Visual Task Adaptation Benchmark (VTAB), which defines good representations as those that adapt to diverse, unseen tasks with few examples. With VTAB, we conduct a large-scale study of many popular publicly-available representation learning algorithms. We carefully control confounders such as architecture and tuning budget. We address questions like: How effective are ImageNet representations beyond standard natural datasets? How do representations trained via generative and discriminative models compare? To what extent can self-supervision replace labels? And, how close are we to general visual representations?

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Evaluation & Benchmarks

Citations in the corpus

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, neutral): “Figure 2 decomposes the VTAB tasks into their respective groups, and compares to previous SOTA methods on this benchmark: BiT, VIVI – a ResNet co-trained on ImageNet and Youtube (Tschannen et al., 2020), and S4L – supervised plus semi-supervised learning on ImageNet (Zhai et al., 2019a).”
On Robustness and Transferability of Convolutional Neural Networks (2020, methodology): “Recent benchmarks have been proposed to evaluate transfer to several datasets, to assess generalization to tasks with different characteristics, or those disjoint from the pre-training data [65, 72].”
Big Transfer (BiT): General Visual Representation Learning (2019, methodology): “These tasks include Im-ageNet’s ILSVRC-2012 [10], CIFAR-10/100 [27], Oxford-IIIT Pet [41], Oxford Flowers-102 [39] (including few-shot variants), and the 1000-sample VTAB-1k benchmark [66], which consists of 19 diverse datasets.”
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, methodology): “We follow the semi-supervised protocol of [74, 76, 8, 32] detailed in Appendix D.”
SLIP: Self-supervision meets Language-Image Pre-training (2021, background): “Prior work on representation learning has argued against end-to-end ﬁnetuning for its sensitivity to optimization hyperparameters [24], and against linear classiﬁcation for being too contrived [63].”
Scaling Vision Transformers (2021, background): “In addition, we report transfer learning result on the VTAB-1k benchmark consisting of 19 tasks [52].”
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, methodology): “For vision models, we utilize our approach to fine-tune a pre-trained ViT [16] for downstream image classification, obtaining superior performance on VTAB-1k [67] benchmark over various image distributions.”
LAION-5B: An open large-scale dataset for training next generation image-text models (2022, methodology): “Additionally, we construct a benchmark we call VTAB+, a superset of VTAB [91], on which we compute the average top-1 accuracy over 35 tasks 10 .”
ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models (2022, background): “2M / 50K 1K WordNet X X VTAB [73] 19 2.”
Visual Prompt Tuning (2022, methodology): “VTAB-1k [80] is a collection of 19 diverse visual classification tasks, which are organized into three groups: Natural - tasks that contain natural images captured using standard cameras; Specialized - tasks that contain images captured via”
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, methodology): “en used for novel tasks, through pre-training [10,11], self-supervision [12,13] or additional annotations [14,15]. However, task adaptation demonstrates that models overt to the data and annotations [16]. Alternatively, one dataset can be enriched with multiple annotations and tasks, towards learnin”
First Session Adaptation: A Strong Replay-Free Baseline for Class-Incremental Learning (2023, methodology): “4.2, we use the 19 datasets of VTAB [42], a low-shot transfer learning benchmark, plus three additional datasets, FGVC-Aircraft [17], Stanford Cars [12], and Letters [6].”
PaLI: A Jointly-Scaled Multilingual Language-Image Model (2022, methodology): “For the VTAB benchmark (Zhai et al., 2019), we follow the methodology outlined in (Zhai et al., 2022b).”