Fluency-Guided Cross-Lingual Image Captioning

By Weiyu Lan, Xirong Li, Jianfeng Dong. 2017

108 citations recorded by the source corpus.

Image captioning has so far been explored mostly in English, as most available datasets are in this language. However, the application of image captioning should not be restricted by language. Only few studies have been conducted for image captioning in a cross-lingual setting. Different from these works that manually build a dataset for a target language, we aim to learn a cross-lingual captioning model fully from machine-translated sentences. To conquer the lack of fluency in the translated sentences, we propose in this paper a fluency-guided learning framework. The framework comprises a module to automatically estimate the fluency of the sentences and another module to utilize the estimated fluency scores to effectively train an image captioning model for the target language. As experiments on two bilingual (English-Chinese) datasets show, our approach improves both fluency and relevance of the generated captions in Chinese, but without using any manually written sentences from the target language.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Ping Luo
3. Ivan Vulic
4. Yu Qiao
5. Mohit Bansal
6. Jifeng Dai
7. Jonas Pfeiffer
8. Tong Lu

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Jianfeng Gao
3. Yu Qiao
4. Xiaogang Wang
5. Dacheng Tao
6. Chunhua Shen
7. Ping Luo
8. Alan L. Yuille

In Multimodal Learning

1. Giuseppe Fiameni
2. Rita Cucchiara
3. Lorenzo Baraldi
4. Marcella Cornia
5. Jianfeng Gao
6. Matteo Stefanini
7. Silvia Cascianelli
8. Ivan Vulic

In Machine Translation

1. Jonas Pfeiffer
2. Elizabeth Clark
3. Siva Reddy
4. Asli Celikyilmaz
5. Ivan Vulic
6. Jianfeng Gao
7. Desmond Elliott

Citations in the corpus

• Evaluation of Text Generation: A Survey (2020, background): “, 2018), image captioning (Lan et al., 2017), video description generation (Park et al.”
• IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages (2022, background): “, 2018) and Mandarin (Lan et al., 2017), but the Mandarin captions do not match the expected test split.”
• From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, background): “ing multilingual captioning systems with unpaired captions [184], [221], [225], [226].”
• Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks (2023, methodology): “In Table 7, we evaluate these capabilities in English using the Flickr30K [88] and COCO [20] datasets, as well as in Chinese using the Flickr30K-CN [58] and COCO-CN [63].”
• SibNet: Sibling Convolutional Encoder for Video Captioning (2018, unknown): “Deep learning-based encoder-decoder architecture [4, 9, 17, 18, 21, 25–27, 31, 34, 41, 48, 52–54, 56] has shown its effectiveness in video captioning.”
• COCO-CN for Cross-Lingual Image Tagging, Captioning, and Retrieval (2018, methodology): “captioning model by exploiting English resources to generate fluent Chinese sentences [14].”