Towards Accurate Text-based Image Captioning with Content Diversity Exploration

By Guanghui Xu, Shuaicheng Niu, Mingkui Tan, Yucheng Luo, Qing Du, Qi Wu. 2021

73 citations recorded by the source corpus.

Text-based image captioning (TextCap) which aims to read and reason images with texts is crucial for a machine to understand a detailed and complex scene environment, considering that texts are omnipresent in daily life. This task, however, is very challenging because an image often contains complex texts and visual information that is hard to be described comprehensively. Existing methods attempt to extend the traditional image captioning methods to solve this task, which focus on describing the overall scene of images by one global caption. This is infeasible because the complex text and visual information cannot be described well within one caption. To resolve this difficulty, we seek to generate multiple captions that accurately describe different parts of an image in detail. To achieve this purpose, there are three key challenges: 1) it is hard to decide which parts of the texts of images to copy or paraphrase; 2) it is non-trivial to capture the complex relationship between diverse texts in an image; 3) how to generate multiple captions with diverse content is still an open problem. To conquer these, we propose a novel Anchor-Captioner method. Specifically, we first find the important tokens which are supposed to be paid more attention to and consider them as anchors. Then, for each chosen anchor, we group its rel-evant texts to construct the corresponding anchor-centred graph (ACG). Last, based on different ACGs, we conduct the multi-view caption generation to improve the content diversity of generated captions. Experimental results show that our method not only achieves SOTA performance but also generates diverse captions to describe images.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Natural Language Generation

Citations in the corpus

GIT: A Generative Image-to-text Transformer for Vision and Language (2022, background): “[113] Guanghui Xu, Shuaicheng Niu, Mingkui Tan, Yucheng Luo, Qing Du, and Qi Wu.”
Efficient Test-Time Model Adaptation without Forgetting (2022, background): “Deep neural networks (DNNs) have achieved excellent performance in many challenging tasks, including image clas-siﬁcation (He et al., 2016; Tan & Le, 2019), video recognition (Wang et al., 2018; Chen et al., 2021), and many other areas (Choi et al., 2018; Fan et al., 2020; Xu et al., 2021).”
From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, background): “Another direction entails generating multiple captions describing different parts of the image, including the contained text [206].”
Multimodal Learning With Transformers: A Survey (2022, background): “graph [228], video to caption [229], [230], [231], [232], image to caption [233], [234], [235], [236], [237], text to speech [238], text to image [205], [239], text to shape [240], RGB to 3D human”
Let there be a clock on the beach: Reducing Object Hallucination in Image Captioning (2021, background): “Image captioning recently shifted into new directions such as generating diverse descriptions [38, 12, 44] by allowing both grounding and controllability [9, 47, 7] while using various contextual information [6, 35].”