CoCa: Contrastive Captioners are Image-Text Foundation Models

By Jiahui Yu, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, Yonghui Wu. 2022

1,785 citations recorded by the source corpus.

Exploring large-scale pretrained foundation models is of significant interest in computer vision because these models can be quickly transferred to many downstream tasks. This paper presents Contrastive Captioner (CoCa), a minimalist design to pretrain an image-text encoder-decoder foundation model jointly with contrastive loss and captioning loss, thereby subsuming model capabilities from contrastive approaches like CLIP and generative methods like SimVLM. In contrast to standard encoder-decoder transformers where all decoder layers attend to encoder outputs, CoCa omits cross-attention in the first half of decoder layers to encode unimodal text representations, and cascades the remaining decoder layers which cross-attend to the image encoder for multimodal image-text representations. We apply a contrastive loss between unimodal image and text embeddings, in addition to a captioning loss on the multimodal decoder outputs which predicts text tokens autoregressively. By sharing the same computational graph, the two training objectives are computed efficiently with minimal overhead. CoCa is pretrained end-to-end and from scratch on both web-scale alt-text data and annotated images by treating all labels simply as text, seamlessly unifying natural language supervision for representation learning. Empirically, CoCa achieves state-of-the-art performance with zero-shot transfer or minimal task-specific adaptation on a broad range of downstream tasks, spanning visual recognition (ImageNet, Kinetics-400/600/700, Moments-in-Time), crossmodal retrieval (MSCOCO, Flickr30K, MSR-VTT), multimodal understanding (VQA, SNLI-VE, NLVR2), and image captioning (MSCOCO, NoCaps). Notably on ImageNet classification, CoCa obtains 86.3% zero-shot top-1 accuracy, 90.6% with a frozen encoder and learned classification head, and new state-of-the-art 91.0% top-1 accuracy on ImageNet with a finetuned encoder.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Multimodal Learning

Citations in the corpus

LAION-5B: An open large-scale dataset for training next generation image-text models (2022, methodology): “Other papers have combined contrastive losses with image captioning to further improve performance [43, 89].”
Scaling Autoregressive Models for Content-Rich Text-to-Image Generation (2022, methodology): “All model variants use the same image tokenizer and CoCa reranking model, described in Section 2.4, with sampling of 16 images per text prompt.”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, methodology): “CoCa [122] studies different model sizes, but on the same pre-training data.”
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (2022, background): “PaLM [11] has 540B parameters, while CoCa [83] has a ≈ 1B parameter text encoder).”
AudioPaLM: A Large Language Model That Can Speak and Listen (2023, neutral): “…[Vaswani et al., 2017] trained to predict sequences of discrete tokens — have been subsequently extended to continuous, natural signals with images [Yu et al., 2022b] or audio waveforms [Lakhotia et al., 2021, Kreuk et al., 2022, Wang et al., 2023] being converted into a stream of discrete…”
Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks (2022, methodology): “Speciﬁcally, our model outperforms previous strong foundation models [YWV + 22, ADL + 22, YCC + 21] despite that we only use public resources for pretraining and ﬁnetuning.”
PaLI: A Jointly-Scaled Multilingual Language-Image Model (2022, methodology): “For context, in prior work, V&L scaling is usually conducted at lower model capacity: for instance, CoCa (Yu et al., 2022) scales up to 2.1B parameters, or scaling is done primarily via the language-modeling backbone, e.g. Flamingo (Alayrac et al., 2022) scales the text backbone to 80B but the image”
Vision-Language Models for Vision Tasks: A Survey (2023, methodology): “The VLM pre-training is usually guided by certain vision-language objectives [10], [18], [19] that enable to learn image-text correspondences from the large-scale image-text pairs [20], [21], e.g. , CLIP [10] employs an image-text contrastive objective and learns by pulling the paired images and…”
Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learning (2022, methodology): “Work on this problem has led to significant improvements along several threads including image level predictions like image classification [13, 54, 58], object level predictions like object detection [33,51,53,59–61], as well as pixel level predictions like semantic segmentation [10,29,51,53].”
MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI (2023, methodology): “Later on, CLIP [60], ALIGN [27], SimVLM [71], CoCa [78], Flamingo [2], BLIP-2 [32], and Fuyu [6] (inter alia) have been proposed to train visual representation using ViT [18] from scratch with massive amount of web data.”
Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone (2022, neutral): “ViLBERT [45], LXMERT [62], UNITER [6] ✓ ✕ ✕ ✓ ✕ ✕ OSCAR [36], VinVL [79] ✓ ✕ ✓ ✓ ✕ ✕ PixelBERT [24], CLIP-ViL [57], ViLT [28] ✓ [32], FLAVA [58], VLMo [66] ✓ ✓ ✕ ✕ ✕ ✓ BLIP [31], CoCa [74], Flamingo [2] ✓ ✓ ✓ ✕ ✕ ✓ MDETR [26], GLIP [34] ✓ ✕ ✕ ✓ ✓ ✓ UniTAB [70], X-VLM [77], OFA [65] ✓ VLP for Vision ”
Sigmoid Loss for Language Image Pre-Training (2023, methodology): “GIT [45], SimVLM [46], and LEMON [21] successfully pre-train models using a generative text decoder instead, while CoCa [52] adds such a decoder to the discriminative CLIP/ALIGN setup, thus combining the pros Algorithm 1 Sigmoid loss pseudo-implementation. and cons of both approaches into a single v”
EVA: Exploring the Limits of Masked Visual Representation Learning at Scale (2022, methodology): “Different from [114, 116] that use multi-head attention pooling and BEiT-3 that exploits an additional pretrained giant language tower as the image classification task layer, we simply adopt a linear layer as the classifier [31].”
Symbolic Discovery of Optimization Algorithms (2023, background): “0% higher than the previous best result (Yu et al., 2022).”
Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers (2024, background): “Training with millions or even billions of image-text pairs [10, 31, 55, 62, 86] has been shown to be effective in learning powerful image foundation models [2, 6, 21, 25, 27, 82].”
ImageBind One Embedding Space to Bind Them All (2023, background): “CoCa [82] adds an image captioning objective on top of the contrastive loss for improved performance.”
MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text (2022, background): “, 2022) and COCA (Yu et al., 2022) proposed end-to-end training for both deep multimodal transformers and unimodal featurization networks and demonstrated”