Visual Clues: Bridging Vision and Language Foundations for Image Paragraph Captioning

By Yujia Xie, Luowei Zhou, Xiyang Dai, Lu Yuan, Nguyen Bach, Ce Liu, Michael Zeng. 2022

31 citations recorded by the source corpus.

People say,"A picture is worth a thousand words". Then how can we get the rich information out of the image? We argue that by using visual clues to bridge large pretrained vision foundation models and language models, we can do so without any extra cross-modal training. Thanks to the strong zero-shot capability of foundation models, we start by constructing a rich semantic representation of the image (e.g., image tags, object attributes / locations, captions) as a structured textual prompt, called visual clues, using a vision foundation model. Based on visual clues, we use large language model to produce a series of comprehensive descriptions for the visual content, which is then verified by the vision model again to select the candidate that aligns best with the image. We evaluate the quality of generated descriptions by quantitative and qualitative measurement. The results demonstrate the effectiveness of such a structured semantic representation.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Zhengyuan Yang
2. Noah A. Smith
3. Jiebo Luo
4. Yushi Hu
5. Hang Hua
6. Weijia Shi
7. Luc Van Gool
8. Cordelia Schmid

In Multimodal Learning

1. Zhengyuan Yang
2. Noah A. Smith
3. Yushi Hu
4. Hang Hua
5. Weijia Shi
6. Jiebo Luo

Citations in the corpus

• PromptCap: Prompt-Guided Task-Aware Image Captioning (2022, background): “Various studies have shown the effectiveness of this framework on multiple tasks, including visual question answering (VQA) [18, 78], image paragraph captioning [75], and video-language tasks [70, 81].”