Adversarial Evaluation of Multimodal Machine Translation

By Desmond Elliott. 2018

95 citations recorded by the source corpus.

The promise of combining language and vision in multimodal machine translation is that systems will produce better translations by leveraging the image data. However, the evidence surrounding whether the images are useful is unconvincing due to inconsistencies between text-similarity metrics and human judgements. We present an adversarial evaluation to directly examine the utility of the image data in this task. Our evaluation tests whether systems perform better when paired with congruent images or incongruent images. This evaluation shows that only one out of three publicly available systems is sensitive to this perturbation of the data. We recommend that multimodal translation systems should be able to pass this sanity check in the future.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, neutral): “Similar problems were found in VQA [3], where simple baselines without sophisticated multimodal understanding performed remarkably well [94, 35, 1, 26] and in multimodal machine translation [18, 74], where the image was found to matter relatively little [13, 17, 7].”
Probing the Need for Visual Context in Multimodal Machine Translation (2019, methodology): “now much more important than previously demonstrated (Elliott, 2018).”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “(Elliott, 2018) found that even in the presence of visual features from unrelated images there is no significant performance losses.”
On Vision Features in Multimodal Machine Translation (2022, background): “Irrelevant images have little impact on the translation quality, and no significant BLEU drop is observed even the image is absent (Elliott, 2018).”
Multimodal Transformer for Multimodal Machine Translation (2020, background): “Inspired by Elliott (2018), we further examine the utility of the image by the adversarial evaluation.”
Looking Enhances Listening: Recovering Missing Speech Using Images (2020, methodology): “Similar to [11], we perform a set of experiments where we misalign images and utterances.”
Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation (2021, background): “Elliott (2018); Gr¨onroos et al. (2018a) provide further evidence by showing that MMT models are, in fact, insensitive to visual input and can translate without signiﬁcant performance losses even in the presence of features derived from unrelated images.”
Neural Machine Translation with Universal Visual Representation (2020, methodology): “, 2019) though the contribution of images is still an open question (Elliott, 2018; Caglayan et al., 2019).”