Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation

By Zhiyong Wu, Lingpeng Kong, W. Bi, Xiang Li, B. Kao. 2021

106 citations recorded by the source corpus.

A neural multimodal machine translation (MMT) system is one that aims to perform better translation by extending conventional text-only translation models with multimodal information. Many recent studies report improvements when equipping their models with the multimodal module, despite the controversy of whether such improvements indeed come from the multimodal part. We revisit the contribution of multimodal information in MMT by devising two interpretable MMT models. To our surprise, although our models replicate similar gains as recently developed multimodal-integrated systems achieved, our models learn to ignore the multimodal information. Upon further investigation, we discover that the improvements achieved by the multimodal models over text-only counterparts are in fact results of the regularization effect. We report empirical findings that highlight the importance of MMT models’ interpretability, and discuss how our findings will benefit future research.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hai Zhao
2. Zhuosheng Zhang
3. Tong Xiao
4. Jingbo Zhu
5. Zuchao Li
6. Rui Wang
7. Zefan Zhou
8. Eiichiro Sumita

In Computer Vision and Pattern Recognition

1. Tong Xiao
2. Jingbo Zhu
3. Zhuosheng Zhang
4. Rui Wang
5. Zefan Zhou
6. Eiichiro Sumita
7. Masso Utiyama
8. Tao Zhou

In Multimodal Learning

1. Rui Wang
2. Tong Xiao
3. Zefan Zhou
4. Eiichiro Sumita
5. Jingbo Zhu
6. Masso Utiyama
7. Tao Zhou
8. Chuanhao Lv

In Machine Translation

1. Rui Wang
2. Eiichiro Sumita

Citations in the corpus

• Multimodal Chain-of-Thought Reasoning in Language Models (2023, neutral): “After obtaining the patch-level vision features, we apply a learnable projection matrix W h to convert the shape of VisionExtractor ( X vision ) into that of H language ; thus we have H vision ∈ R m × d where m is the number of patches.”
• Universal Multimodal Representation for Language Understanding (2023, result): “The results of existing methods are from [68].”
• On Vision Features in Multimodal Machine Translation (2022, background): “To address the above issue, Yao and Wan (2020) proposed a multimodal self-attention to consider the relative difference of information between two modalities.”