Unpaired Image Captioning by Language Pivoting

By Jiuxiang Gu, Shafiq R. Joty, Jianfei Cai, G. Wang. 2018

89 citations recorded by the source corpus.

Image captioning is a multimodal task involving computer vision and natural language processing, where the goal is to learn a mapping from the image to its natural language description. In general, the mapping function is learned from a training set of image-caption pairs. However, for some language, large scale image-caption paired corpus might not be available. We present an approach to this unpaired image captioning problem by language pivoting. Our method can effectively capture the characteristics of an image captioner from the pivot language (Chinese) and align it to the target language (English) using another pivot-target (Chinese-English) parallel corpus. We evaluate our method on two image-to-English benchmark datasets: MSCOCO and Flickr30K. Quantitative comparisons against several baseline approaches demonstrate the effectiveness of our method.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, methodology): “We also compare VisualGPT against unsupervised meth-ods of Gu et al. [22] and Feng et al. [20], which use tens of millions of unpaired images and captions.”
Learning to Collocate Neural Modules for Image Captioning (2019, background): “Compared with them, modern image captioners which achieve superior performances are attention based encoder-decoder methods [53, 52, 43, 6, 60, 34, 2, 55, 35, 36, 56, 41, 12, 13].”
Unsupervised Image Captioning (2018, methodology): “To further demonstrate the effectiveness of the proposed three objectives, we compare with [15] under the same unpaired captioning setting, where the COCO captions of the training images are used but in an unpaired manner.”
Improving Captioning for Low-Resource Languages by Cycle Consistency (2019, background): “The ﬁrst kind of approach is based on translation [12–14].”
Image Captioning with Very Scarce Supervised Data: Adversarial Semi-Supervised Learning Approach (2019, background): “Most of aforementioned works [Gu et al. , 2018; Anne Hendricks et al. , 2016; Venugopalan et al. , 2017; Feng et al. , 2019] exploit large auxiliary supervised data which is often beyond image-caption data.”
Scene Graph Generation With External Knowledge and Image Reconstruction (2019, background): “Scene graph provides a structured representation of an image that can support a wide range of high-level visual tasks, including image captioning [12, 14, 13, 43], visual question answering [36, 38, 47], image retrieval [11, 21], and image generation [20].”
From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, background): “work [184] proposes to generate captions in a pivot language and then translate predicted captions to the target language.”