More Grounded Image Captioning by Distilling Image-Text Matching Model

By Yuanen Zhou, Meng Wang, Daqing Liu, Zhenzhen Hu, Hanwang Zhang. 2020

148 citations recorded by the source corpus.

Visual attention not only improves the performance of image captioners, but also serves as a visual interpretation to qualitatively measure the caption rationality and model transparency. Specifically, we expect that a captioner can fix its attentive gaze on the correct objects while generating the corresponding words. This ability is also known as grounded image captioning. However, the grounding accuracy of existing captioners is far from satisfactory.To improve the grounding accuracy while retaining the captioning quality, it is expensive to collect the word-region alignment as strong supervision.To this end, we propose a Part-of-Speech (POS) enhanced image-text matching model (SCAN): POS-SCAN, as the effective knowledge distillation for more grounded image captioning. The benefits are two-fold: 1) given a sentence and an image, POS-SCAN can ground the objects more accurately than SCAN; 2) POS-SCAN serves as a word-region alignment regularization for the captioner's visual attention module. By showing benchmark experimental results, we demonstrate that conventional image captioners equipped with POS-SCAN can significantly improve the grounding accuracy without strong supervision. Last but not the least, we explore the indispensable Self-Critical Sequence Training (SCST) in the context of grounded image captioning and show that the image-text matching score can serve as a reward for more grounded captioning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhe Gan
2. Lijuan Wang
3. Zicheng Liu
4. Jianfeng Wang
5. Xiaowei Hu
6. Jianfeng Gao
7. Zhengyuan Yang
8. Linjie Li

In Computer Vision and Pattern Recognition

1. Zhe Gan
2. Lijuan Wang
3. Zicheng Liu
4. Jianfeng Gao
5. Jianfeng Wang
6. Xiaowei Hu
7. Zhengyuan Yang
8. Luc Van Gool

In Multimodal Learning

1. Lijuan Wang
2. Zhe Gan
3. Zicheng Liu
4. Jianfeng Wang
5. Xiaowei Hu
6. Zhengyuan Yang
7. Jianfeng Gao
8. Yumao Lu

In Natural Language Generation

1. Jianfeng Gao
2. Zhengyuan Yang
3. Hanwang Zhang
4. Ron Mokady
5. Amir Hertz

Citations in the corpus

• VIVO: Surpassing Human Performance in Novel Object Captioning with Visual Vocabulary Pre-Training (2020, methodology): “2019; Li, Jiang, and Han 2019), grounded captioning (Ma et al. 2020; Zhou et al. 2020b), image captioning with reading comprehension (Sidorov et al.”
• Crossing the Format Boundary of Text and Boxes: Towards Unified Vision-Language Modeling (2021, methodology): “For grounded captioning [38, 70, 72], we benchmark on the Flickr30k Entities dataset [42].”
• Deconfounded Image Captioning: A Causal Retrospect (2020, methodology): “SAT [30] was the first work which applied the attention mechanism in IC, and then this mechanism was used in almost every follow-up IC systems [24], [25], [36], [37], [38], [39], [40], [41].”
• iPerceive: Applying Common-Sense Reasoning to Multi-Modal Dense Video Captioning and Video Question Answering (2020, background): “The union of these fields has been explored in several contexts, including image classification [34, 4], reinforcement learning [39], adversarial learning [26], visual dialog [44], image captioning [65] and scene/knowledge graph generation [51, 40].”
• Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic (2023, unknown): “Tasks with input boxes: Given an image and a location box, the task of Grounding Caption (GC) (Zhou et al., 2020) is to generate a description for this location by considering the surrounding environment.”