Learning Visual Representations with Caption Annotations

By Mert Bulent Sariyildiz, J. Perez, Diane Larlus. 2020

173 citations recorded by the source corpus.

Pretraining general-purpose visual features has become a crucial part of tackling many computer vision tasks. While one can learn such features on the extensively-annotated ImageNet dataset, recent approaches have looked at ways to allow for noisy, fewer, or even no annotations to perform such pretraining. Starting from the observation that captioned images are easily crawlable, we argue that this overlooked source of information can be exploited to supervise the training of visual representations. To do so, motivated by the recent progresses in language models, we introduce {\em image-conditioned masked language modeling} (ICMLM) -- a proxy task to learn visual representations over image-caption pairs. ICMLM consists in predicting masked words in captions by relying on visual cues. To tackle this task, we propose hybrid models, with dedicated visual and textual encoders, and we show that the visual representations learned as a by-product of solving this task transfer well to a variety of target tasks. Our experiments confirm that image captions can be leveraged to inject global and localized semantic information into visual representations. Project website: this https URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “Adopting more recent architectures and pre-training approaches, VirTex (Desai & Johnson, 2020), ICMLM (Bulent Sariyildiz et al., 2020), and ConVIRT (Zhang et al., 2020) have recently demonstrated the potential of transformer-based language modeling, masked language modeling, and contrastive…”
Hierarchical Text-Conditional Image Generation with CLIP Latents (2022, background): “Recent progress in computer vision has been driven by scaling models on large datasets of captioned images collected from the internet [10, 44, 60, 39, 31, 16].”
Contrastive Learning of Medical Visual Representations from Paired Images and Text (2020, background): “Several concurrent papers have studied the problem of learning visual representations from text data (Sariyildiz et al., 2020; Desai and Johnson, 2021) on general-domain image problems.”
VirTex: Learning Visual Representations from Textual Annotations (2020, methodology): “[75] on learning visual representations from captions via image conditioned masked language modeling, with one major difference – we train our entire model from scratch, whereas they rely on pretrained BERT for textual features.”
SLIP: Self-supervision meets Language-Image Pre-training (2021, background): “ICMLM [48] and VirTex [15] showed that language supervision on COCO Captions produced useful visual representations.”
StyleGAN-NADA (2021, background): “Typically, to solve these tasks, a cross-modal vision and language representation is learned [10, 13, 25–27, 30, 42,47,48], often by training a transformer [52].”
StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery (2021, background): “Joint representations Multiple works learn cross-modal Vision and language (VL) representations [8, 39, 44, 29, 24, 43, 23, 4, 26] for a variety of tasks, such as languagebased image retrieval, image captioning, and visual question answering.”
Combined Scaling for Zero-shot Transfer Learning (2021, methodology): “Using images with accompanying text is related to the literature on image-captioning models, such as (Vinyals et al., 2015; Karpathy and Fei-Fei, 2015; Xu et al., 2015; Joulin et al., 2016; Li et al., 2017; Sariyildiz et al., 2020; Zhang et al., 2020; Desai and Johnson, 2021).”
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision (2021, background): “These works are however limited to small datasets such as Flickr (Joulin et al., 2015; Li et al., 2017) and COCO Captions (Desai & Johnson, 2020; Sariyildiz et al., 2020), and the resulting models don’t produce a vision-language representation that is needed for tasks like cross-modal retrieval.”
Unified Contrastive Learning in Image-Text-Label Space (2022, background): “(ii) Another line of works focuses on learning transferable visual representation from natural language supervisions, including generative [12, 48] and contrastive methods [16,29,47,59,60,74].”
RegionCLIP: Region-based Language-Image Pretraining (2021, background): “The most relevant work is learning from natural language, such as image tags [3, 8, 12, 25, 28] and text descriptions [11, 23, 43, 53, 61].”
Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning (2021, background): “These models are designed either for specific vision-language task [17] or vision task [9, 32].”
Support-set bottlenecks for video-text representation learning (2020, background): “…of transformers in the NLP domain, several works have leveraged transformers to learn transferable image (Chen et al., 2020a; Desai & Johnson, 2020; Sariyildiz et al., 2020) or multi-modal image-text representations (Chen et al., 2019; Li et al., 2020; 2019; Lu et al., 2019; Su et al., 2019; Tan &…”
Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm (2021, background): “Language supervision Joulin et al. (2016); Gomez et al. (2017); Zhang et al. (2020); Sariyildiz et al. (2020); Desai & Johnson (2021) demonstrate the effectiveness of learning transferable visual features from language supervision.”
Robust fine-tuning of zero-shot models (2021, background): “One increasingly popular class of vision models are those pre-trained with auxiliary language supervision, which can be used for zero-shot inference [18, 44, 75, 79, 84, 107, 109].”
Exploring Visual Engagement Signals for Representation Learning (2021, methodology): “VirTex and ICMLM achieve similar results as VisE with less training data (118k vs. {615k, 1.23M}).”
DenseCLIP: Extract Free Dense Labels from CLIP (2021, background): “Some studies [12, 16, 17, 44, 47, 53] pro-pose to learn from image-caption pairs.”
Prompt Distribution Learning (2022, background): “A lot of approaches attempt to learn representations by predicting the captions of images [8, 19, 23, 35, 47].”