Multimodal Transformer for Multimodal Machine Translation

By Shaowei Yao, Xiaojun Wan. 2020

194 citations recorded by the source corpus.

Multimodal Machine Translation (MMT) aims to introduce information from other modality, generally static images, to improve the translation quality. Previous works propose various incorporation methods, but most of them do not consider the relative importance of multiple modalities. Equally treating all modalities may encode too much useless information from less important modalities. In this paper, we introduce the multimodal self-attention in Transformer to solve the issues above in MMT. The proposed method learns the representation of images based on the text, which avoids encoding irrelevant information in images. Experiments and visualization analysis demonstrate that our model benefits from visual information and substantially outperforms previous works and competitive baselines in terms of various metrics.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Model Architectures, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Jianfeng Gao
  3. Zhe Gan
  4. Jingjing Liu
  5. Linjie Li
  6. Yu Cheng
  7. Ivan Vulic
  8. Yejin Choi

In Computer Vision and Pattern Recognition

  1. Mohit Bansal
  2. Zhe Gan
  3. Linjie Li
  4. Luc Van Gool
  5. Jingjing Liu
  6. Dacheng Tao
  7. Cordelia Schmid
  8. Yu Cheng

In Multimodal Learning

  1. Mohit Bansal
  2. Linjie Li
  3. Jingjing Liu
  4. Zhe Gan
  5. Yu Cheng
  6. Jaemin Cho
  7. Jie Lei
  8. Lijuan Wang

In Model Architectures

  1. Jonas Pfeiffer
  2. Mohit Bansal
  3. Yu Cheng
  4. Siva Reddy
  5. Ivan Vulic
  6. Luowei Zhou
  7. Jaemin Cho
  8. Shuohang Wang

In Machine Translation

  1. Jonas Pfeiffer
  2. Jingjing Liu
  3. Linjie Li
  4. Shuohang Wang
  5. Mohit Bansal
  6. Siva Reddy
  7. Ivan Vulic
  8. Jaemin Cho

Citations in the corpus