VirTex: Learning Visual Representations from Textual Annotations

By Karan Desai, Justin Johnson. 2020

484 citations recorded by the source corpus.

The de-facto approach to many vision tasks is to start from pretrained visual representations, typically learned via supervised training on ImageNet. Recent methods have explored unsupervised pretraining to scale to vast quantities of unlabeled images. In contrast, we aim to learn high-quality visual representations from fewer images. To this end we revisit supervised pretraining, and seek data-efficient alternatives to classification-based pretraining. We propose VirTex – a pretraining approach using semantically dense captions to learn visual representations. We train convolutional networks from scratch on COCO Captions, and transfer them to downstream recognition tasks including image classification, object detection, and instance segmentation. On all tasks, VirTex yields features that match or exceed those learned on ImageNet – supervised or unsupervised – despite using up to ten times fewer images.

Subfields: Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “Our initial approach, similar to VirTex, jointly trained an image CNN and text transformer from scratch to predict the caption of an image.”
Hierarchical Text-Conditional Image Generation with CLIP Latents (2022, background): “Recent progress in computer vision has been driven by scaling models on large datasets of captioned images collected from the internet [10, 44, 60, 39, 31, 16].”
Contrastive Learning of Medical Visual Representations from Paired Images and Text (2020, background): “Several concurrent papers have studied the problem of learning visual representations from text data (Sariyildiz et al., 2020; Desai and Johnson, 2021) on general-domain image problems.”
Towards Language-Free Training for Text-to-Image Generation (2021, background): “Unlike the aforementioned works, another line of works focus on the way of learning visual representation from natural language supervisions, including both generative [5] and discriminative [48, 49, 58] methods.”
Projected GANs Converge Faster (2021, methodology): “We evaluate CLIP-FID (using a ResNet50 trained with the CLIP objective on the dataset collected by [60]) and VirTex-FID (using a ResNet50 trained on COCO Captions with the VirTex objective [11]).”
Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, background): “Cross-attention itself is in widespread use as a mechanism for allowing information from a source to influence processing of a target domain of a different size or structure [18, 20, 53, 88, 64, 68, 34, 51].”
SLIP: Self-supervision meets Language-Image Pre-training (2021, background): “ICMLM [48] and VirTex [15] showed that language supervision on COCO Captions produced useful visual representations.”
How Much Can CLIP Benefit Vision-and-Language Tasks? (2021, background): “CLIP (Contrastive Language-Image Pre-training) (Radford et al., 2021) 3 falls into the line of research that learns visual representations from natural language supervision (Desai and Johnson, 2020; Sariyildiz et al., 2020; Jia et al., 2021).”
StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery (2021, background): “Joint representations Multiple works learn cross-modal Vision and language (VL) representations [8, 39, 44, 29, 24, 43, 23, 4, 26] for a variety of tasks, such as languagebased image retrieval, image captioning, and visual question answering.”
Flamingo: a Visual Language Model for Few-Shot Learning (2022, methodology): “In our ablation studies (Section 3.3), we compare the proposed GATED XATTN - DENSE layers against recent alternatives [22, 68] and explore the effect of how frequently these additional layers are inserted to trade off between efficiency and expressivity.”
Combined Scaling for Zero-shot Transfer Learning (2021, methodology): “Using images with accompanying text is related to the literature on image-captioning models, such as (Vinyals et al., 2015; Karpathy and Fei-Fei, 2015; Xu et al., 2015; Joulin et al., 2016; Li et al., 2017; Sariyildiz et al., 2020; Zhang et al., 2020; Desai and Johnson, 2021).”
LiT: Zero-Shot Transfer with Locked-image text Tuning (2021, background): “Another approach, which we adopt in this work, is to learn an alignment between image and text embedding spaces [6, 15, 21, 22, 31, 70].”
Learning to Prompt for Vision-Language Models (2021, background): “Matching images and text features has been formulated as metric learning (Frome et al., 2013), multi-label classiﬁcation (Joulin et al., 2016; Gomez et al., 2017), n-gram language learning (Li et al., 2017), and the recently proposed captioning (Desai and Johnson, 2021).”
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision (2021, background): “These works are however limited to small datasets such as Flickr (Joulin et al., 2015; Li et al., 2017) and COCO Captions (Desai & Johnson, 2020; Sariyildiz et al., 2020), and the resulting models don’t produce a vision-language representation that is needed for tasks like cross-modal retrieval.”
Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “Several recent methods [5, 15, 18, 32, 45, 47, 49, 53, 54, 70, 73, 98] pretrain multi-modal vision-language representations, such as transformers, using datasets with image captions, e.”