Improving Captioning for Low-Resource Languages by Cycle Consistency

By Yike Wu, Shiwan Zhao, Jia Chen, Ying Zhang, Xiaojie Yuan, Zhong Su. 2019

8 citations recorded by the source corpus.

Improving the captioning performance on low-resource languages by leveraging English caption datasets has received increasing research interest in recent years. Existing works mainly fall into two categories: translation-based and alignment-based approaches. In this paper, we propose to combine the merits of both approaches in one unified architecture. Specifically, we use a pre-trained English caption model to generate high-quality English captions, and then take both the image and generated English captions to generate low-resource language captions. We improve the captioning performance by adding the cycle consistency constraint on the cycle of image regions, English words, and low-resource language words. Moreover, our architecture has a flexible design which enables it to benefit from large monolingual English caption datasets. Experimental results demonstrate that our approach outperforms the state-of-the-art methods on common evaluation metrics. The attention visualization also shows that the proposed approach really improves the fine-grained alignment between words and image regions.

Subfields: Computer Vision and Pattern Recognition, Machine Translation, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Kai Yi
2. Boyang Albert Li
3. Han Guo
4. Mohamed Elhoseiny
5. Jun Chen
6. Tinne Tuytelaars
7. Yu Qiao
8. Dacheng Tao

Citations in the corpus

• VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, background): “Data for specialized domains like medical report generation [15,40] and low-resource language captioning [18,74] cannot be easily scaled.”