STAIR Captions: Constructing a Large-Scale Japanese Image Caption Dataset

By Yuya Yoshikawa, Yutaro Shigeto, Akikazu Takeuchi. 2017

129 citations recorded by the source corpus.

In recent years, automatic generation of image descriptions (captions), that is, image captioning, has attracted a great deal of attention. In this paper, we particularly consider generating Japanese captions for images. Since most available caption datasets have been constructed for English language, there are few datasets for Japanese. To tackle this problem, we construct a large-scale Japanese image caption dataset based on images from MS-COCO, which is called STAIR Captions. STAIR Captions consists of 820,310 Japanese captions for 164,062 images. In the experiment, we show that a neural network trained using STAIR Captions can generate more natural and better Japanese captions, compared to those generated using English-Japanese machine translation after generating English captions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Evaluation & Benchmarks

Citations in the corpus

MURAL: Multimodal, Multitask Retrieval Across Languages (2021, methodology): “The STAIR dataset (Yoshikawa et al., 2017) adds human crowd-sourced Japaneses captions for MSCOCO images.”
Captioning Images Taken by People Who Are Blind (2020, methodology): “The recent trend has been to include a larger number of examples, relying on scraping images from the web (typically Flickr) to support the growth from a few thousand [18, 19, 40] to hundreds of thousands [15, 25, 30] of captioned images in such datasets.”
Emergent Translation in Multi-Agent Communication (2017, methodology): “We take the German corpus from Multi30k Task2, the Japanese corpus from STAIR, and train two models on a sentence-level communication task between the two languages.”
COCO-CN for Cross-Lingual Image Tagging, Captioning, and Retrieval (2018, background): “on MS-COCO that target Japanese [10], [17], German”
How2: A Large-scale Dataset for Multimodal Language Understanding (2018, unknown): “Task Dataset Languages Audio Visual Size IC Flickr8K [19] EN, TR [20], ZH [21] 7 3 (I) 8K IC Flickr30K [22] 150K EN, DE [23] 7 3 (I) 30K IC MSCOCO [6] 414K EN 7 3 (I) 82K IC 820K JA [24] 7 3 (I) 164K MMT Multi30K [23] EN, DE, FR [25], CZ [26] 7 3 (I) 30K VD MSVD [27] 122K total in many 3 3 (V) 5.”
M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training (2021, methodology): “STAIR Captions [34] extended MSCOCO [2] with 820K Japanese captions for COCO images.”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, methodology): “The STAIR5 dataset consists of Japanese captions (Yoshikawa et al., 2017) for the 164,062 MSCOCO images.”
Sheffield MultiMT: Using Object Posterior Predictions for Multimodal Machine Translation (2017, background): “Recent attempts have been made to incorporate multilinguality into both these large-scale datasets, with the datasets being extended to other languages such as German and Japanese (Elliott et al., 2016; Miyazaki and Shimizu, 2016; Yoshikawa et al., 2017).”
Models of Visually Grounded Speech Signal Pay Attention to Nouns: A Bilingual Experiment on English and Japanese (2019, unknown): “Using the same methodology as [2], [6] collected 5 Japanese captions for each image of the original MSCOCO dataset.”
LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval (2021, methodology): “, 2018) with captions in German, French and Czech; and COCO Japanese (Yoshikawa et al., 2017) and Chinese (Li et al.”
Image Pivoting for Learning Multilingual Multimodal Representations (2017, unknown): “Recently there have been attempts to create image descriptions and models for other languages (Funaki and Nakayama, 2015; Elliott et al., 2016; Rajendran et al., 2016; Miyazaki and Shimizu, 2016; Specia et al., 2016; Li et al., 2016; Hitschler et al., 2016; Yoshikawa et al., 2017).”
M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training (2020, methodology): “STAIR Captions[33] extended MSCOCO [16] with 820K Japanese captions for COCO images.”
Universal Multimodal Representation for Language Understanding (2023, background): “As a result, the visual information is only applied to the translation task over speciﬁc multimodal datasets [25, 26, 27, 28, 29], instead of general text-only NMT [30, 31, 32] and low-resource text-only NMT [33, 34, 35, 36].”
Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description (2017, background): “…work on creating multilingual image description datasets in German (Elliott et al., 2016; Hitschler et al., 2016; Rajendran et al., 2016), Turkish (Unal et al., 2016), Chinese (Li et al., 2016), Japanese (Miyazaki and Shimizu, 2016; Yoshikawa et al., 2017), and Dutch (van Miltenburg et al., 2017).”
UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training (2021, methodology): “STAIR [51] extends the original MSCOCO dataset by collecting 820K Japanese captions for 165K COCO images.”