Incorporating Global Visual Features into Attention-based Neural Machine Translation.

By Iacer Calixto, Qun Liu. 2017

167 citations recorded by the source corpus.

We introduce multi-modal, attention-based neural machine translation (NMT) models which incorporate visual features into different parts of both the encoder and the decoder. Global image features are extracted using a pre-trained convolutional neural network and are incorporated (i) as words in the source sentence, (ii) to initialise the encoder hidden state, and (iii) as additional data to initialise the decoder hidden state. In our experiments, we evaluate translations into English and German, how different strategies to incorporate global image features compare and which ones perform best. We also study the impact that adding synthetic multi-modal, multilingual data brings and find that the additional data have a positive impact on multi-modal NMT models. We report new state-of-the-art results and our best models also significantly improve on a comparable phrase-based Statistical MT (PBSMT) model trained on the Multi30k data set according to all metrics evaluated. To the best of our knowledge, it is the first time a purely neural model significantly improves over a PBSMT model on all metrics evaluated on this data set.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Machine Translation

Citations in the corpus

VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research (2019, background): “Additionally, other studies [39, 12, 40, 9] explore the cross-modal feature fusion of images and sentences.”
Probing the Need for Visual Context in Multimodal Machine Translation (2019, methodology): “Finally, we also experiment with encoder-decoder initialization (INIT) (Calixto and Liu, 2017; Caglayan et al., 2017a) where we initialize both the encoder and the decoder using a non-linear transformation of the pool5 features.”
Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description (2017, methodology): “…al., 2017) CMU Carnegie Melon University (Jaffe, 2017) CUNI Univerzita Karlova v Praze (Helcl and Libovick´y, 2017) DCU-ADAPT Dublin City University (Calixto et al., 2017a) LIUMCVC Laboratoire d’Informatique de l’Universit´e du Maine & Universitat Autonoma de Barcelona Computer Vision Center…”
Unsupervised Multimodal Neural Machine Translation with Pseudo Visual Pivoting (2020, unknown): “Huang et al. (2016) encode word sequences with regional visual objects while Calixto and Liu (2017) leverage global visual feature.”
Multimodal Transformer for Multimodal Machine Translation (2020, methodology): “Calixto and Liu (2017) utilize global image features to initialize the encoder/decoder hidden states of RNN. Elliott and K´ad´ar (2017) model the source sentence and reconstruct the image representation jointly via multi-task learning.”
On Vision Features in Multimodal Machine Translation (2022, methodology): “Early attempts mainly focused on enhancing the MMT model by better incorporation of the vision features (Calixto and Liu, 2017; Elliott and Kádár, 2017; Delbrouck and Dupont, 2017).”
Exploiting Cross-Sentence Context for Neural Machine Translation (2017, methodology): “Calixto et al. (2017) utilize global image features extracted using a pre-trained convolutional neural network and incorporate them in NMT.”
UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training (2021, background): “Early works on cross-lingual multimodal tasks mainly focus on machine translation [22, 55, 6, 50, 2] and imagetext retrieval [18, 28, 5, 19, 49].”