PromptCap: Prompt-Guided Task-Aware Image Captioning

By Yushi Hu, Hang Hua, Zhengyuan Yang, Weijia Shi, Noah A. Smith, Jiebo Luo. 2022

142 citations recorded by the source corpus.

Knowledge-based visual question answering (VQA) involves questions that require world knowledge beyond the image to yield the correct answer. Large language models (LMs) like GPT-3 are particularly helpful for this task because of their strong knowledge retrieval and reasoning capabilities. To enable LM to understand images, prior work uses a captioning model to convert images into text. However, when summarizing an image in a single caption sentence, which visual entities to describe are often underspecified. Generic image captions often miss visual details essential for the LM to answer visual questions correctly. To address this challenge, we propose PromptCap (Prompt-guided image Captioning), a captioning model designed to serve as a better connector between images and black-box LMs. Different from generic captions, PromptCap takes a natural-language prompt to control the visual entities to describe in the generated caption. The prompt contains a question that the caption should aid in answering. To avoid extra annotation, PromptCap is trained by examples synthesized with GPT-3 and existing datasets. We demonstrate PromptCap's effectiveness on an existing pipeline in which GPT-3 is prompted with image captions to carry out VQA. PromptCap outperforms generic captions by a large margin and achieves state-of-the-art accuracy on knowledge-based VQA tasks (60.4% on OK-VQA and 59.6% on A-OKVQA). Zero-shot results on WebQA show that PromptCap generalizes well to unseen domains.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

ViperGPT: Visual Inference via Python Execution for Reasoning (2023, neutral): “TRiG [13] 50.5 KAT [16] 54.4 RA-VQA [32] 54.5 REVIVE [33] 58.0 PromptCap [21] 58.8 Z S PNP-VQA [52] 35.9 PICa [60] 43.3 BLIP-2 [30] 45.9 Flamingo [1] 50.6 ViperGPT (ours) 51.9”
REPLUG: Retrieval-Augmented Black-Box Language Models (2023, unknown): “, 2020) and open-domain question answering (Lewis et al., 2020; Izacard et al., 2022b; Hu et al., 2022).”
LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding (2023, background): “We leave generating more detailed and knowledge-enhanced captions [38] for future work.”
Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models (2024, background): “With the advancement of LMs [4, 35, 42, 45, 13], researchers have demonstrated the possiblity of decomposing complex vision tasks into simpler substeps that can each be solved using vision tools [57, 60, 18, 17].”
BLINK: Multimodal Large Language Models Can See but Not Perceive (2024, background): “There have been extended studies that explore further capabilities of multimodal LLMs, especially on VQA reasoning [30–32,38,39,66, 81,89].”
VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use (2023, methodology): “For each instruction, we next collect instruction-conditioned captions – unlike prior work [23, 24] these descriptions are designed not only to describe the image in general, but also, surface information targeted to the instruction.”
TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering (2023, unknown): “This benchmark also comes with different VQA models [57, 28, 58, 32, 31, 21] that can be used to evaluate generative models and can be easily extended to use future VQA models when they become available.”