ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic

By Yoad Tewel, Yoav Shalev, Idan Schwartz, Lior Wolf. 2021

260 citations recorded by the source corpus.

Recent text-to-image matching models apply contrastive learning to large corpora of uncurated pairs of images and sentences. While such models can provide a powerful score for matching and subsequent zero-shot tasks, they are not capable of generating caption given an image. In this work, we repurpose such models to generate a descriptive text given an image at inference time, without any further training or tuning step. This is done by combining the visual-semantic model with a large language model, benefiting from the knowledge in both web-scale models. The resulting captions are much less restrictive than those obtained by supervised captioning methods. Moreover, as a zero-shot learning method, it is extremely flexible and we demonstrate its ability to perform image arithmetic in which the inputs can be either images or text and the output is a sentence. This enables novel high-level vision capabilities such as comparing two images or solving visual analogy tests. Our code is available at: https://github.com/YoadTew/zero-shot-image-to-text.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, background): “SMs substantially outperform the zero-shot state-of-the-art ZeroCap [62] with a CIDEr [64] score 18.”
InstructPix2Pix: Learning to Follow Image Editing Instructions (2022, methodology): “Techniques for combining pretrained models include joint finetuning on a new task [4, 33, 40, 67], communication through prompting [62,69], composing probability distributions of energy-based models [11, 37], guiding one model with feedback from another [61], and iterative optimization [34].”
Language Models are General-Purpose Interfaces (2022, methodology): “COCO Caption Karpathy Test Model BLEU-4 CIDEr METEOR SPICE ZeroCap (Tewel et al., 2021) 2.”
WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentation (2023, background): “Other works presented CLIP’s promise in zero-/few-shot transfer to downstream tasks beyond classification [11,30,41,46].”
Composing Ensembles of Pre-trained Models via Iterative Consensus (2022, methodology): “To integrate feedback from scorers to the generator, imilar to (Tewel et al., 2021), we define a context cache Ct (a set of embedding functions in GPT-2) that stores the context information generated so far, which is updated iteratively based on the feedback from scorers.”
Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunities (2024, background): “Although primarily focusing on processing and understanding textual information, some LLM models also have remarkable image processing capabilities in vision-related tasks [136], including image-to-text generation [137] and object detection [138], etc.”
Language Models Can See: Plugging Visual Controls in Text Generation (2022, methodology): “The closest to our proposal is ZeroCap [78] which is an unsupervised image captioning method by combining frozen CLIP and GPT-2.”