Findings of the Third Shared Task on Multimodal Machine Translation

By Loïc Barrault, Fethi Bougares, Lucia Specia, Chiraag Lala, Desmond Elliott, Stella Frank. 2018

160 citations recorded by the source corpus.

We present the results from the third shared task on multimodal machine translation. In this task a source sentence in English is supplemented by an image and participating systems are required to generate a translation for such a sentence into German, French or Czech. The image can be used in addition to (or instead of) the source sentence. This year the task was extended with a third target language (Czech) and a new test set. In addition, a variant of this task was introduced with its own test set where the source sentence is given in multiple languages: English, French and German, and participating systems are required to generate a translation in Czech. Seven teams submitted 45 different systems to the two variants of the task. Compared to last year, the performance of the multimodal submissions improved, but text-only systems remain competitive.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Machine Translation

Citations in the corpus

MURAL: Multimodal, Multitask Retrieval Across Languages (2021, background): “Elliott et al. (2017) and Barrault et al. (2018) further extend the dataset by collecting human translations of English Flickr30k captions to French and Czech.”
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision (2021, methodology): “We test the multilingual model on Multi30k, a multilingual image text retrieval dataset extends Flickr30K (Plum-mer et al., 2015) to German (de) (Elliott et al., 2016), French (fr) (Elliott et al., 2017) and Czech (cs) (Barrault et al., 2018).”
VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research (2019, background): “Previous studies mainly focus on using images as the visual modality to help machine translation [54, 19, 6].”
Findings of the 2018 Conference on Machine Translation (WMT18) (2018, background): “…shared tasks are described in separate papers in these proceedings: • biomedical translation (Neves et al., 2018), • multimodal machine translation (Barrault et al., 2018), • metrics (Ma et al., 2018), • quality estimation (Specia et al., 2018), • automatic post-editing (Chatterjee et…”
Probing the Need for Visual Context in Multimodal Machine Translation (2019, methodology): “Czech and German.”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, methodology): “The aim of MMT (Specia et al., 2016; Hitschler et al., 2016; Elliott et al., 2017; Barrault et al., 2018) is to translate source language sentences that describe an image into target language.”
IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages (2022, background): “Texts in most multimodal datasets are usually only available in English, or in high-resource languages (e.g., Chinese and a few major Indo-European languages; Elliott et al., 2016; 2017; Barrault et al., 2018).”
Learning to Scale Multilingual Representations for Vision-Language Tasks (2020, methodology): “is task, the goal is to retrieve a relevant sentence from a database given an image or to retrieve a relevant image from a database given a sentence. We augment current multilingual datasets Multi30K [6,14,15,43] and MSCOCO [27,28,31] using machine translations so that every image has at least ve se”
Multimodal Transformer for Multimodal Machine Translation (2020, background): “It initially organized as a shared task within the First Conference on Machine Translation (Specia et al., 2016; Elliott et al., 2017; Bar-rault et al., 2018).”
LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval (2021, methodology): “We consider two benchmarks: Multi30K (Elliott et al., 2016, 2017; Barrault et al., 2018) with cap-tions in German, French and Czech; and COCO Japanese (Yoshikawa et al., 2017) and Chinese (Li et al., 2019b).”
Neural Machine Translation with Universal Visual Representation (2020, methodology): “, 2017; Barrault et al., 2018). Subsequently, there has been a rise in the number of studies (Caglayan et al., 2016; 2017; Calixto et al., 2016; Huang et al., 2016; Libovickỳ & Helcl, 2017; Helcl et al., 2018). For example, Calixto et al. (2017) proposed a doubly-attentive multi-modal NMT model to i”
Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrieval (2021, methodology): “For multilingual experiments, we use the standard Multi30k dataset (Elliott et al., 2016, 2017; Barrault et al., 2018), which extends Flickr30k with 5 German and one French and Czech caption per image.”
UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training (2021, methodology): “The presentation of the result is simpli-ﬁed to only include the Meta-Average for the mutilingual image-text retrieval over both Multi30K and MSCOCO, the accuracy on VQA v2.”
Neural Machine Translation: A Review (2019, background): “Multimodality in translation was the subject of a series of WMT shared tasks (Barrault et al., 2018; Elliott et al., 2017; Specia et al., 2016).”
Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation (2021, background): “In particular, Specia et al. (2016); Elliott et al. (2017); Barrault et al. (2018) argue that visual context does not seem to help translation reliably, at least as measured by automatic metrics.”
Visual Grounding in Video for Unsupervised Word Translation (2020, background): “Of particular interest, recent work has focused on multimodal and multilingual settings such as producing captions in many languages, visual-guided translations [8, 16, 44, 46], or bilingual visual question answering [18].”
How2: A Large-scale Dataset for Multimodal Language Understanding (2018, methodology): “So far, MMT has been addressed using captioning datasets extended with translations in different languages such as IAPR-TC12 [38] and Multi30K which is an extension of Flickr30K into German [20], French [22], and Czech [23].”