Multimodal Transformer for Multimodal Machine Translation
By Shaowei Yao, Xiaojun Wan. 2020
194 citations recorded by the source corpus.
Multimodal Machine Translation (MMT) aims to introduce information from other modality, generally static images, to improve the translation quality. Previous works propose various incorporation methods, but most of them do not consider the relative importance of multiple modalities. Equally treating all modalities may encode too much useless information from less important modalities. In this paper, we introduce the multimodal self-attention in Transformer to solve the issues above in MMT. The proposed method learns the representation of images based on the text, which avoids encoding irrelevant information in images. Experiments and visualization analysis demonstrate that our model benefits from visual information and substantially outperforms previous works and competitive baselines in terms of various metrics.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Model Architectures, Machine Translation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
In Multimodal Learning
In Model Architectures
In Machine Translation
Citations in the corpus
- Unifying Vision-and-Language Tasks via Text Generation (2021, methodology): “We compare our method with state-of-the-art transformer models: Multimodal self-attention (MSA) (Yao & Wan, 2020), MeMAD (Gr¨onroos et al., 2018).”
- UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training (2021, background): “Early works on cross-lingual multimodal tasks mainly focus on machine translation [22, 55, 6, 50, 2] and image-text retrieval [18, 28, 5, 19, 49].”
- Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation (2021, unknown): “To this end, Yao and Wan (2020) and Yin et al. (2020) replace the vanilla Transformer encoder with a multi-modal encoder.”
- IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages (2022, methodology): “…et al., 2021) and additional modalities (Armitage et al., 2020), we have recently witnessed pioneering work on multimodal machine translation (Yao & Wan, 2020; Huang et al., 2020), multilingual visual question answering (Pfeiffer et al., 2022), multilingual text-to-video search (Huang et…”