Injecting Semantic Concepts into End-to-End Image Captioning

By Zhiyuan Fang, Jianfeng Wang, Xiaowei Hu, Lin Liang, Zhe Gan, Lijuan Wang, Yezhou Yang, Zicheng Liu. 2021

127 citations recorded by the source corpus.

Tremendous progresses have been made in recent years in developing better image captioning models, yet most of them rely on a separate object detector to extract regional features. Recent vision-language studies are shifting towards the detector-free trend by leveraging grid representations for more flexible model training and faster inference speed. However, such development is primarily focused on image understanding tasks, and remains less investigated for the caption generation task. In this paper, we are concerned with a better-performing detector-free image captioning model, and propose a pure vision transformer-based image captioning model, dubbed as ViTCAP, in which grid representations are used without extracting the regional features. For improved performance, we introduce a novel Concept Token Network (CTN) to predict the semantic concepts and then incorporate them into the end-to-end captioning. In particular, the CTN is built on the basis of a vision transformer, and is designed to predict the concept tokens through a classification task, from which the rich semantic information contained greatly benefits the captioning task. Compared with the previous detector-based models, ViTCAP drastically simplifies the architectures and at the same time achieves competitive performance on various challenging image captioning datasets. In particular, ViTCAP reaches 138.1 CIDEr scores on COCO-caption Karpathy-split, 93.8 and 108.6 CIDEr scores on nocaps and Google-CC captioning datasets, respectively.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhe Gan
2. Jianfeng Gao
3. Lijuan Wang
4. Zicheng Liu
5. Jianfeng Wang
6. Linjie Li
7. Xiaowei Hu
8. Zhengyuan Yang

In Computer Vision and Pattern Recognition

1. Zhe Gan
2. Jianfeng Gao
3. Lijuan Wang
4. Zicheng Liu
5. Luc Van Gool
6. Cordelia Schmid
7. Dacheng Tao
8. Jianfeng Wang

In Multimodal Learning

1. Zhe Gan
2. Lijuan Wang
3. Zicheng Liu
4. Jianfeng Wang
5. Xiaowei Hu
6. Linjie Li
7. Jianfeng Gao
8. Zhengyuan Yang

In Natural Language Generation

1. Jianfeng Gao
2. Zhengyuan Yang

Citations in the corpus

• GIT: A Generative Image-to-text Transformer for Vision and Language (2022, methodology): “Recently, a growing interest is in dense representation (Huang et al., 2020; Wang et al., 2021b;a; Kim et al., 2021; Fang et al., 2021b; Dou et al., 2021; Li et al., 2021a) from the feature map, which requires no bounding box annotations.”
• Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone (2022, methodology): “Models trained with CIDEr optimization ViTCAP [17] 4M 41.”
• VC-GPT: Visual Conditioned GPT for End-to-End Generative Vision-and-Language Pre-training (2022, methodology): “The probability of the token in the t th position is determined by all the past tokens and the visual information.”