COCO-CN for Cross-Lingual Image Tagging, Captioning, and Retrieval

By Xirong Li, Chaoxi Xu, Xiaoxu Wang, Weiyu Lan, Zhengxiong Jia, Gang Yang, Jieping Xu. 2018

186 citations recorded by the source corpus.

This paper contributes to cross-lingual image annotation and retrieval in terms of data and baseline methods. We propose COCO-CN, a novel dataset enriching MS-COCO with manually written Chinese sentences and tags. For effective annotation acquisition, we develop a recommendation-assisted collective annotation system, automatically providing an annotator with several tags and sentences deemed to be relevant with respect to the pictorial content. Having 20 342 images annotated with 27 218 Chinese sentences and 70 993 tags, COCO-CN is currently the largest Chinese–English dataset that provides a unified and challenging platform for cross-lingual image tagging, captioning, and retrieval. We develop conceptually simple yet effective methods per task for learning from cross-lingual resources. Extensive experiments on the three tasks justify the viability of the proposed dataset and methods. Data and code are publicly available at https://github.com/li-xirong/coco-cn.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

MURAL: Multimodal, Multitask Retrieval Across Languages (2021, methodology): “†: Numbers from UC2 paper; these were fine-tuned on MSCOCO-CN (Li et al., 2019), which has a different split than en and ja, resulting in possible train/test infiltration.”
WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning (2021, background): “Consequently there exists only a handful of non-English data collections such as Multi30K-DE (German) [11], DeCOCO (German) [13], Multi30K-FR (French) [10], Multi30K-CS (Czech) [3], YJCaptions26k (Japanese) [23] and MS-COCO-CN (Chinese) [20].”
Universal Multimodal Representation for Language Understanding (2023, background): “As a result, the visual information is only applied to the translation task over specific multimodal datasets [25], [26], [27], [28], [29], instead of general text-only NMT [30], [31], [32] and lowresource text-only NMT [33], [34], [35], [36].”
M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training (2020, methodology): “As for Chinese, we use the COCO-CN split [31].”
Towards Zero-shot Cross-lingual Image Retrieval (2020, background): “Datasets like (Yoshikawa et al., 2017; Li et al., 2018; Elliott et al., 2016; Grubinger et al., 2006; Wu et al., 2017) provide data with images for only 2-3 languages which do not help in scaling models to a large number of languages.”
M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training (2021, methodology): “Multi30K [35] 32K 384K 5 MSCOCO [2] [34] [20] 120K 1.”
LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval (2021, methodology): “Similarly, Li et al. (2019b) collected 1-2 Chinese captions per image for 20K COCO images to build COCO Chinese.”
UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training (2021, methodology): “For MSCOCO Chinese, we follow the original split as in [34].”
From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, background): “Chinese and Japanese captions for COCO images [222], [223], German captions for Flick30K [224]), or directly train-”
Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks (2023, methodology): “In Table 7, we evaluate these capabilities in English using the Flickr30K [88] and COCO [20] datasets, as well as in Chinese using the Flickr30K-CN [58] and COCO-CN [63].”
Learning to Scale Multilingual Representations for Vision-Language Tasks (2020, methodology): “We augment current multilingual datasets Multi30K [5,12,13,41] and MSCOCO [26,25,29] using machine translations so that every image has at least five sentences across ten diverse languages: English (En), German (De), French (Fr), Czech (Cs), Chinese (Cn), Japanese (Ja), Arabic (Ar), Afrikaans (Af), ”
IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages (2022, methodology): “Remarkably, this tendency is less accentuated in MaRVL, where pretraining Mandarin multimodal data is still insufﬁcient to tackle the out-of-distribution nature of MaRVL’s culture-speciﬁc concepts (Table 15 in Appendix C.2).”