ClipCap: CLIP Prefix for Image Captioning

By Ron Mokady, Amir Hertz. 2021

863 citations recorded by the source corpus.

Image captioning is a fundamental task in vision-language understanding, where the model predicts a textual informative caption to a given input image. In this paper, we present a simple approach to address this task. We use CLIP encoding as a prefix to the caption, by employing a simple mapping network, and then fine-tunes a language model to generate the image captions. The recently proposed CLIP model contains rich semantic features which were trained with textual context, making it best for vision-language perception. Our key idea is that together with a pre-trained language model (GPT2), we obtain a wide understanding of both visual and textual data. Hence, our approach only requires rather quick training to produce a competent captioning model. Without additional annotations or pre-training, it efficiently generates meaningful captions for large-scale and diverse datasets. Surprisingly, our method works well even when only the mapping network is trained, while both CLIP and the language model remain frozen, allowing a lighter architecture with less trainable parameters. Through quantitative evaluation, we demonstrate our model achieves comparable results to state-of-the-art methods on the challenging Conceptual Captions and nocaps datasets, while it is simpler, faster, and lighter. Our code is available in https://github.com/rmokady/CLIP_prefix_caption.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, methodology): “5, but do not perform as well as methods such as ClipCap [45] which are directly finetuned on the training set.”
Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “One recent line of work [26, 68, 78, 114, 136, 144] proposes to freeze the pretrained LM weights to prevent catastrophic forgetting [71].”
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, methodology): “4.1 Following Stanford Alpaca [43], we utilize 52K instruction-following data for training, which is extended from 175 instruction-output pairs [45].”
InstructPix2Pix: Learning to Follow Image Editing Instructions (2022, methodology): “Techniques for combining pretrained models include joint finetuning on a new task [4, 33, 40, 67], communication through prompting [62,69], composing probability distributions of energy-based models [11, 37], guiding one model with feedback from another [61], and iterative optimization [34].”
LAION-5B: An open large-scale dataset for training next generation image-text models (2022, methodology): “Beyond image classification and retrieval, the community later adapted CLIP to further vision tasks such as object navigation and visual question answering [17, 32, 50, 72].”
KNN-Diffusion: Image Generation via Large-Scale Retrieval (2022, methodology): “The joint representation was shown to hold a strong semantic alignment between the two modalities, enabling image generation (Liu et al., 2021; Wang et al., 2022), image manipulation (Patashnik et al., 2021; Avrahami et al., 2022b), and image captioning (Mokady et al., 2021).”
Language Models Can See: Plugging Visual Controls in Text Generation (2022, methodology): “Recently, vision-language pre-training methods [94, 40, 51, 26], benefiting from the rich visual-textual representation of pre-trained models on large-scale datasets, are tendencies for vision-language generation by re-training or fine-tuning the model parameters on downstream tasks.”
Transfer Visual Prompt Generator across LLMs (2023, methodology): “In fact, the VPG and projector trained on LLM src have already learned how to map the visual content to LLM src ’s understandable soft prompt [39].”
Pengi: An Audio Language Model for Audio Tasks (2023, methodology): “Each prefix embedding is mapped to the highest similarity token from the GPT2 vocabulary [44].”
RemoteCLIP: A Vision Language Foundation Model for Remote Sensing (2023, background): “Beyond classification, CLIP models also exhibit the flexibility to be integrated into a variety of vision-language applications, including detection [33], [34], segmentation [35], [36], captioning [37], [38], VQA [39], [40], and conditional image generation [41].”
A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge (2022, methodology): “ClipCap [33] (row p) is an image captioning method that passes CLIP image features through a trained network to GPT-2 (as input tokens).”
RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning (2022, background): “One of the most popular scheme is to tune soft prompts (i.e., continuous embedding vectors) as they are amenable to gradient descent (Li and Liang, 2021; Vu et al., 2021; Gu et al., 2021; Liu et al., 2021d; Mokady et al., 2021; Qian et al., 2022; An et al., 2022, etc.).”
MAPL: Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Prompting (2022, background): “3.1), containing only 3.4M parameters – orders of magnitude fewer than ClipCap’s (43M) and Flamingo’s (194M).”
Zero-Shot Video Question Answering via Frozen Bidirectional Language Models (2022, background): “Recently, a promising line of work builds on frozen large autoregressive language models [22, 75, 102, 107, 116, 123] for zero-shot visual question answering.”
ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic (2021, methodology): “We compared our results with ClipCap [50] that encodes the image with CLIP’s image encoder and uses it as an initial token for GPT-2.”
Cross-lingual and Multilingual CLIP (2022, positive): “…main contribution was image classification without predefined labels, CLIP representations have since proved useful for various other tasks such as Image-Text retrieval (Radford et al., 2021), Visual Question-Answering (Shen et al., 2021), and Automatic Image Captioning (Mokady et al., 2021).”
From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, neutral): “subsequent captioning approaches [95], [96], [97].”
Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers (2024, neutral): “2) Soft prompts often fall short of interpretability (Khashabi et al., 2021; Lester et al., 2021b; Hambardzumyan et al., 2021; Mokady et al., 2021).”
CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Alignment (2022, background): “, 2022), image-text generation (Patashnik et al., 2021; Mokady et al., 2021) and so on (Guzhov et al.”