The MeMAD Submission to the WMT18 Multimodal Translation Task

By Stig-Arne Grönroos, Benoit Huet, Mikko Kurimo, Jorma T. Laaksonen, Bernard Merialdo, Phu Pham, Mats Sjöberg, Umut Sulubacak, Jörg Tiedemann, Raphael Troncy and 1 more. 2018

69 citations recorded by the source corpus.

This paper describes the MeMAD project entry to the WMT Multimodal Machine Translation Shared Task. We propose adapting the Transformer neural machine translation (NMT) architecture to a multi-modal setting. In this paper, we also describe the preliminary experiments with text-only translation systems leading us up to this choice. We have the top scoring system for both English-to-German and English-to-French, according to the automatic metrics for flickr18. Our experiments show that the effect of the visual features in our system is small. Our largest gains come from the quality of the underlying text-only NMT system. We find that appropriate use of additional data is effective.

Subfields: Multimodal Learning, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Multimodal Learning

In Machine Translation

Citations in the corpus

Unifying Vision-and-Language Tasks via Text Generation (2021, methodology): “We compare our method with state-of-the-art transformer models: Multimodal self-attention (MSA) (Yao & Wan, 2020), MeMAD (Gr¨onroos et al., 2018).”
Probing the Need for Visual Context in Multimodal Machine Translation (2019, background): “Nevertheless, the conclusion about the contribution of the visual modality is still unclear: Grönroos et al. (2018) consider their multimodal gains “modest” and attribute the largest gain to the usage of external parallel corpora.”
Universal Multimodal Representation for Language Understanding (2023, result): “Compared with the results in text-only NMT, we ﬁnd that the image enhancement sometimes gives marginal contribution, which is consistent with the ﬁndings in previous work [77, 78, 79].”
On Vision Features in Multimodal Machine Translation (2022, methodology): “For training, we used Adam Op-timizer (Kingma and Ba, 2015) with β 1 = 0 .”
Neural Machine Translation with Universal Visual Representation (2020, background): “, 2017) or only marginally beneficial (Grönroos et al., 2018).”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, unknown): “1 MeMAD (Grönroos et al., 2018) METEOR 56.”
Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation (2021, methodology): “From the table, we see that although we can replicate similar BLEU scores of Transformer-Base as reported in (Gr¨onroos et al., 2018b; Ive et al., 2019), these scores (Row 1) are signiﬁcantly outperformed by Transformer-Small and Transformer-Tiny, which have fewer parameters.”