Visual to Text: Survey of Image and Video Captioning

By Sheng Li, Zhiqiang Tao, Kang Li, Yun Raymond Fu. 2019

106 citations recorded by the source corpus.

Visual data such as images and videos are easily accessible nowadays, and they play critical roles in many real-world applications like surveillance. This raises a series of technological demands for automatic visual understanding and content summarization, which has guided the research community to move towards a better achievement of such capabilities. Meanwhile, it presents the big challenge of semantic understanding of video content and automatically translating them into human language. When developing such automatic translation systems, one critical issue is how to bridge the gap between low level features and high level semantic information. Furthermore, as a large amount of videos are captured under unconstrained conditions by nonprofessional users, this issue becomes even more serious. Therefore, brand new sets of technologies are required to address these difficulties and narrow the semantic gap effectively. These thoughts drive us to survey the complete state-of-the-art techniques in the visual to text topic. Existing methods, popular datasets, technical difficulties, and promising future directions are discussed systematically. In particular, we classify existing methods by their mechanism to link visual information (including both images and videos) and text descriptions, and emphasize the latest advances on deep learning based approaches. The quantitative evaluations of representative approaches on benchmark dataset are also presented and discussed. Finally, we provide with the promising research directions on this topic.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Natural Language Generation

Citations in the corpus

Multimodal Few-Shot Learning with Frozen Language Models (2021, methodology): “Following standard captioning systems [22, 13], we treat captioning as conditional generation of caption text y given an image x.”
RATT: Recurrent Attention to Transient Tasks for Continual Image Captioning (2020, methodology): “For more details on image captioning we refer to recent surveys [14, 18].”
Evaluation of Text Generation: A Survey (2020, background): “Others survey specific NLG tasks or NLG models, such as image captioning (Kilickaya et al., 2017; Hossain et al., 2018; Li et al., 2019; Bai & An, 2018), machine translation (Dabre et al., 2020; Han & Wong, 2016; Wong & Kit, 2019), summarization (Deriu et al., 2009; Shi et al., 2018), question…”
Concadia: Tackling image accessibility with context (2021, result): “This is well-suited to our task since it is a simple architecture, optimized for interpretability, but still achieves comparable results with state-of-the-art models (see Li et al. 2019; Bai and An 2018; Jing et al. 2020; Hossain et al. 2019; Liu et al. 2019 for large-scale comparative analyses).”