A Shared Task on Multimodal Machine Translation and Crosslingual Image Description

By Lucia Specia, Stella Frank, Khalil Sima'an, Desmond Elliott. 2016

255 citations recorded by the source corpus.

This paper introduces and summarises the findings of a new shared task at the intersection of Natural Language Processing and Computer Vision: the generation of image descriptions in a target language, given an image and/or one or more descriptions in a different (source) language. This challenge was organised along with the Conference on Machine Translation (WMT16), and called for system submissions for two task variants: (i) a translation task, in which a source language image description needs to be translated to a target language, (optionally) with additional cues from the corresponding image, and (ii) a description generation task, in which a target language description needs to be generated for an image, (optionally) with additional cues from source language descriptions of the same image. In this first edition of the shared task, 16 systems were submitted for the translation task and seven for the image description task, from a total of 10 teams.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Learned in Translation: Contextualized Word Vectors (2017, methodology): “Our smallest MT dataset comes from the WMT 2016 multi-modal translation shared task [Specia et al., 2016].”
Findings of the 2016 Conference on Machine Translation (2016, background): “, 2016) • multimodal machine translation and crosslingual image description (Specia et al., 2016) • bilingual document alignment (Buck and Koehn, 2016)”
The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, background): “…88, 43, 68, 28], with offshoot tasks focusing speciﬁcally on visual reasoning [36], referring expressions [39], visual storytelling [57, 32], visual dialogue [9, 12], multimodal machine translation [18, 74], visual reasoning [75, 33, 71, 85, 27], visual common sense reasoning [91] and many others.”
VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research (2019, background): “Previous studies mainly focus on using images as the visual modality to help machine translation [54, 19, 6].”
Multi30K: Multilingual English-German Image Descriptions (2016, methodology): “For more examples of approaches targeting these applications, we refer the reader to the forthcoming report on the WMT16 shared task on Multimodal Machine Translation and Crosslingual Image Description (Specia et al., 2016).”
Findings of the Third Shared Task on Multimodal Machine Translation (2018, background): “This was followed by a wide range of work in the ﬁrst two editions of this shared task at the WMT in 2016 and 2017 (Specia et al., 2016; Elliott et al., 2017).”
Probing the Need for Visual Context in Multimodal Machine Translation (2019, methodology): “Initially organized as a shared task within the First Conference on Machine Translation (WMT16) (Specia et al., 2016), MMT has so far been studied using the Multi30K dataset (Elliott et al., 2016), a multilingual extension of Flickr30K (Young et al., 2014) with translations of the English image desc”
Adversarial Evaluation of Multimodal Machine Translation (2018, methodology): “In this paper, we use the Meteor text-similarity score (Denkowski and Lavie, 2014) because it naturally decomposes to the sentence level, and it is already the de-facto evaluation metric for multimodal machine translation (Specia et al., 2016).”
Imagination Improves Multimodal Translation (2017, methodology): “Multimodal machine translation is the task of translating sentences in context, such as images paired with a parallel text (Specia et al., 2016).”
Unsupervised Multimodal Neural Machine Translation with Pseudo Visual Pivoting (2020, unknown): “Multimodal machine translation (Specia et al., 2016) (MMT) considers additional images as a complementary information source for MT.”
LIUM-CVC Submissions for WMT17 Multimodal Translation Task (2017, background): “Although our submission ranked first among multimodal systems for CIC task, it was not able to improve over purely textual NMT baselines in neither tasks (Specia et al., 2016).”
Image Pivoting for Learning Multilingual Multimodal Representations (2017, unknown): “Recently there have been attempts to create image descriptions and models for other languages (Funaki and Nakayama, 2015; Elliott et al., 2016; Rajendran et al., 2016; Miyazaki and Shimizu, 2016; Specia et al., 2016; Li et al., 2016; Hitschler et al., 2016; Yoshikawa et al., 2017).”
How2: A Large-scale Dataset for Multimodal Language Understanding (2018, background): “From a computational perspective, major breakthroughs in natural language processing (NLP), computer vision (CV), and automatic speech recognition (ASR) have resulted in improvements in a wide range of multimodal tasks, including visual question-answering [2], multimodal machine translation [3], vis”