Multi30K: Multilingual English-German Image Descriptions

By Desmond Elliott, Stella Frank, Khalil Sima'an, Lucia Specia. 2016

660 citations recorded by the source corpus.

We introduce the Multi30K dataset to stimulate multilingual multimodal research. Recent advances in image description have been demonstrated on English-language datasets almost exclusively, but image description should not be limited to English. This dataset extends the Flickr30K dataset with i) German translations created by professional translators over a subset of the English descriptions, and ii) descriptions crowdsourced independently of the original English descriptions. We outline how the data can be used for multilingual image description and multimodal machine translation, but we anticipate the data will be useful for a broader range of tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Evaluation & Benchmarks

Citations in the corpus

Experience Grounds Language (2020, background): “Elliott et al. (2016) takes the classic problem of machine translation and reframes it in the context of visual observations (Elliott and Kádár, 2017).”
Pretrained Transformers Improve Out-of-Distribution Robustness (2020, background): “We (Lang, 1995), the English source side of English-German WMT16 and English-German Multi30K (Elliott et al., 2016), and concatenations of the premise and hypothesis for RTE (Dagan et al., 2005) and SNLI (Bowman et al., 2015).”
The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, background): “Similar problems were found in VQA [3], where simple baselines without sophisticated multimodal understanding performed remarkably well [94, 35, 1, 26] and in multimodal machine translation [18, 74], where the image was found to matter relatively little [13, 17, 7].”
MURAL: Multimodal, Multitask Retrieval Across Languages (2021, background): “5 Translating a text query with 10 tokens adds additional latency of upto 400ms in run on CPU with a batch size of 1, Multi30k and MSCOCO.”
SimVLM: Simple Visual Language Model Pretraining with Weak Supervision (2021, methodology): “After pretrained, our model is finetuned and evaluated on six vision-language benchmarks, including three discriminative tasks: VQA v2 (Goyal et al., 2017), SNLI-VE (Xie et al., 2019), and NLVR2 (Suhr et al., 2018); as well as three generative tasks: CoCo captioning (Chen et al., 2015), NoCaps (Agra”
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision (2021, methodology): “We test the multilingual model on Multi30k, a multilingual image text retrieval dataset extends Flickr30K (Plum-mer et al., 2015) to German (de) (Elliott et al., 2016), French (fr) (Elliott et al., 2017) and Czech (cs) (Barrault et al., 2018).”
Unifying Vision-and-Language Tasks via Text Generation (2021, methodology): “This might be because the source text in Multi30K contains sufﬁcient information for translation as discussed in Caglayan et al. (2019)”
VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research (2019, methodology): “The Multi30K dataset [20], which is annotated based on the image captioning dataset Flickr30K [44], is commonly used in this direction.”
Unsupervised Machine Translation Using Monolingual Corpora Only (2017, background): “The task 1 of the Multi30k dataset (Elliott et al., 2016) has 30,000 images, with annotations in English, French and German, that are translations of each other.”
WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning (2021, background): “Consequently there exists only a handful of non-English data collections such as Multi30K-DE (German) [11], DeCOCO (German) [13], Multi30K-FR (French) [10], Multi30K-CS (Czech) [3], YJCaptions26k (Japanese) [23] and MS-COCO-CN (Chinese) [20].”
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, background): “There have also been a number of shared tasks in this area, including the coco (‘Common Objects in Context’) Captioning Challenge(6), organised as part of the Large-Scale Scene Understanding Challenge (lsun)(7) and the Multimodal Machine Translation Task (Elliott et al., 2016).”
Learning to Deceive with Attention-Based Explanations (2019, methodology): “We use the Multi30K dataset, comprising of image descriptions (Elliott et al., 2016).”
PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification (2019, background): “Existing multilingual datasets for paraphrase iden-tiﬁcation, e.g. Multi30k (Elliott et al., 2016) and Opusparcus (Creutz, 2018), lack challenging examples like PAWS.”
Findings of the Third Shared Task on Multimodal Machine Translation (2018, background): “The Multi30K dataset (Elliott et al., 2016) is the primary resource for the shared task.”
Attention-based Multimodal Neural Machine Translation (2016, methodology): “In the ofﬁcial WMT 2016 multimodal translation task dataset (Elliott et al., 2016), there are 29,000 parallel sentences from English to German for training, 1014 for validation and 1000 for testing.”
Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO (2020, background): “Image captioning data sets such as Flickr8k (Rashtchian et al., 2010), Flickr30k (Young et al., 2014), its multi-lingual extension Multi30k (Elliott et al., 2016), MSCOCO (Lin et al., 2014), and Conceptual Cap-tions (Sharma et al., 2018) are incomplete because the only known relationships are betwee”
Probing the Need for Visual Context in Multimodal Machine Translation (2019, methodology): “…task within the First Conference on Machine Translation (WMT16) (Specia et al., 2016), MMT has so far been studied using the Multi30K dataset (Elliott et al., 2016), a multilingual extension of Flickr30K (Young et al., 2014) with translations of the English image descriptions into German,…”
Are BLEU and Meaning Representation in Opposition? (2018, methodology): “The de models were trained on the Multi30K multilingual image caption dataset (Elliott et al., 2016), extended by Helcl and Libovick´y (2017b), who acquired additional parallel data using back-translation (Sennrich et al., 2016) and perplexity-based selection (Yasuda et al., 2008).”