Probing the Need for Visual Context in Multimodal Machine Translation

By Ozan Caglayan, Pranava Swaroop Madhyastha, Lucia Specia, Loïc Barrault. 2019

158 citations recorded by the source corpus.

Current work on multimodal machine translation (MMT) has suggested that the visual modality is either unnecessary or only marginally beneficial. We posit that this is a consequence of the very simple, short and repetitive sentences used in the only available dataset for the task (Multi30K), rendering the source text sufficient as context. In the general case, however, we believe that it is possible to combine visual and textual information in order to ground translations. In this paper we probe the contribution of the visual modality to state-of-the-art MMT models by conducting a systematic analysis where we partially deprive the models from source-side textual context. Our results show that under limited textual context, models are capable of leveraging the visual input to generate better translations. This contradicts the current belief that MMT models disregard the visual modality because of either the quality of the image features or the way they are integrated into the model.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, neutral): “Similar problems were found in VQA [3], where simple baselines without sophisticated multimodal understanding performed remarkably well [94, 35, 1, 26] and in multimodal machine translation [18, 74], where the image was found to matter relatively little [13, 17, 7].”
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts (2021, background): “The same pheonmenon has been reported for multi-modal machine translation, where models trained on image+text tend to ignore the image and primarily use the text input [8].”
VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research (2019, methodology): “Therefore, we conduct noun/verb masking experiments [11] to investigate to what extent the video information can help machine translation.”
Don’t Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases (2019, background): “For example, many joint natu-1 github.com/chrisc36/debias ral language processing and vision datasets can be partially solved by models that ignore the vision aspect of the task (Jabri et al., 2016; Zhang et al., 2016; Anand et al., 2018; Caglayan et al., 2019).”
Revisiting the Evaluation of Theory of Mind through Question Answering (2019, background): “Similarly, natural language inference benchmarks have been shown to be vulnerable to bias exploitation (Gururangan et al., 2018; Poliak et al., 2018), and multimodal machine translation benchmarks have been demonstrated to be too simple to require multi-modality (Caglayan et al., 2019).”
Unsupervised Multimodal Neural Machine Translation with Pseudo Visual Pivoting (2020, unknown): “As analyzed in (Caglayan et al., 2019), visual content is more critical when the textual content is limited or uncertain in MMT.”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “Other studies (Caglayan et al., 2019) have reduced the criticism of MMT by showing that under the limited textual context, MMT models are capable of leveraging the visual input to generate better translations.”
On Vision Features in Multimodal Machine Translation (2022, methodology): “In this work, we address this question by a systematic study of using various vision models in MMT, in particular using the most successful models in recent studies (such as Vision Transformer, or ViT for short).”
Can Automatic Post-Editing Improve NMT? (2020, background): “Sentence-level APE cannot address these issues robustly, which calls for further research on multimodal (Deena et al., 2017; Caglayan et al., 2019) and document-level (Hard-”
Looking Enhances Listening: Recovering Missing Speech Using Images (2020, methodology): “Silence Masking: We substitute the masked word with a specific value, silence, similar to the special token in [12].”
Neural Machine Translation with Universal Visual Representation (2020, background): “Compared with the results in text-only NMT, we find that the image presentation gave marginal contribution, which was consistent with the findings in previous work (Zhang et al., 2017; Grönroos et al., 2018; Caglayan et al., 2019).”
Universal Multimodal Representation for Language Understanding (2023, background): “The most plausible reason might be that the sentences in Multi30K are quite simple, short, and repetitive, so that the source text itself is sufficient to perform the translation [10], [79].”
Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation (2021, methodology): “We conduct experiment with a new masking strategy that does not need any entity linking annotations as in Caglayan et al. (2019).”
The Importance of Modeling Social Factors of Language: Theory and Practice (2021, background): “Several works have shown, though, how incorporating non-textual information can improve performance, speciﬁcally in conjunction with images (Lazaridou et al., 2015; Caglayan et al., 2019).”