Using Artificial Tokens to Control Languages for Multilingual Image Caption Generation

By Satoshi Tsutsui, David J. Crandall. 2017

23 citations recorded by the source corpus.

Recent work in computer vision has yielded impressive results in automatically describing images with natural language. Most of these systems generate captions in a sin- gle language, requiring multiple language-specific models to build a multilingual captioning system. We propose a very simple technique to build a single unified model across languages, using artificial tokens to control the language, making the captioning system more compact. We evaluate our approach on generating English and Japanese captions, and show that a typical neural captioning architecture is capable of learning a single model that can switch between two different languages.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. William Yang Wang
2. Xin Eric Wang
3. Yuan-fang Wang
4. Jiawei Wu
5. Gang Yang
6. Chaoxi Xu
7. Lei Li
8. Zhengxiong Jia

In Natural Language Processing

1. Xin Eric Wang
2. Lei Li
3. Gang Yang
4. William Yang Wang
5. Yuan-fang Wang
6. Weiyu Lan
7. Chaoxi Xu
8. Jiawei Wu

In Multimodal Learning

1. Xin Eric Wang
2. William Yang Wang
3. Gang Yang
4. Chaoxi Xu
5. Lei Li
6. Yuan-fang Wang
7. Weiyu Lan
8. Jiawei Wu

Citations in the corpus

• VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research (2019, background): “Meanwhile, multilingual image captioning is introduced to describe the content of an image with multiple languages [32, 57, 33].”
• COCO-CN for Cross-Lingual Image Tagging, Captioning, and Retrieval (2018, background): “However, only recently has the topic gained increasing aŠention [6, 7, 13, 16, 24, 26, 28].”