Attention-based Multimodal Neural Machine Translation

By Po-Yao (Bernie) Huang, Frederick Liu, Sz-Rung Shiang, Jean Oh, Chris Dyer. 2016

183 citations recorded by the source corpus.

We present a novel neural machine translation (NMT) architecture associating visual and textual features for translation tasks with multiple modalities. Transformed global and regional visual features are concatenated with text to form attendable sequences which are dissipated over parallel long short-term memory (LSTM) threads to assist the encoder generating a representation for attention-based decoding. Experiments show that the proposed NMT outperform the text-only baseline.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “Languages where large swathes of monolingual data are not easily available might require more sample efﬁ-cient approaches for language modeling, including grounding with visual modalities or sources of information (Huang et al., 2016), and learning from meta-data or other forms of context.”
RobustFill: Neural Program Learning under Noisy I/O (2017, methodology): “Double attention is a straightforward extension to the standard attentional architecture, similar to the multimodal attention described in Huang et al. (2016).”
A Shared Task on Multimodal Machine Translation and Crosslingual Image Description (2016, background): “CMU+NTU Carnegie Melon University (Huang et al., 2016) CUNI Univerzita Karlova v Praze (Libovický et al.”
Probing the Need for Visual Context in Multimodal Machine Translation (2019, background): “, 2017) and (iii) the integration of regional features from object detection networks (Huang et al., 2016; Grönroos et al., 2018).”
Evaluating Discourse Phenomena in Neural Machine Translation (2017, methodology): “Inspired by work on multi-modal translation (Caglayan et al., 2016; Huang et al., 2016), multi-encoder translation models have recently been used to incorporate extra-sentential linguistic context in purely textual NMT (Zoph and Knight, 2016; Libovick´y and Helcl, 2017; Wang et al., 2017).”
Neural Machine Translation with Universal Visual Representation (2020, background): “Subsequently, there has been a rise in the number of studies (Caglayan et al., 2016; 2017; Calixto et al., 2016; Huang et al., 2016; Libovickỳ & Helcl, 2017; Helcl et al., 2018).”
Using Images to Improve Machine-Translating E-Commerce Product Listings. (2017, methodology): “, 2016), different research groups have proposed to include images in re-ranking n-best lists generated by an SMT system or directly in a NMT framework with some success (Caglayan et al., 2016; Calixto et al., 2016; Huang et al., 2016; Libovický et al., 2016; Shah et al., 2016).”
Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation (2021, background): “It can also be appended/prepended to word embeddings as additional input tokens (Huang et al., 2016; Calixto and Liu, 2017).”
Visually Grounded Word Embeddings and Richer Visual Features for Improving Multimodal Neural Machine Translation (2017, background): “A few attempts [4, 5, 6, 7, 8] have been made during the WMT 2016 Multimodal Machine Translation evaluation campaign.”
Unsupervised Multimodal Neural Machine Translation with Pseudo Visual Pivoting (2020, unknown): “Huang et al. (2016) encode word sequences with regional visual objects while Calixto and Liu (2017) leverage global visual feature.”
An empirical study on the effectiveness of images in Multimodal Neural Machine Translation (2017, methodology): “The best results of NMT model were those of Huang et al. (2016) who used LSTM fed with global visual features or multiple regional visual features followed by rescoring.”
Sheffield MultiMT: Using Object Posterior Predictions for Multimodal Machine Translation (2017, methodology): “Recent approaches to Multimodal NMT have used low level image features, including dense fully connected vectors and spatial convolutional representations from an image classification network (Elliott et al., 2015; Huang et al., 2016).”
Neural Machine Translation: A Review (2019, methodology): “using global image features extracted with a separate computer vision model [555, 556] or via visual attention [557].”
Attention Forcing for Sequence-to-sequence Model Training (2019, methodology): “Examples include BLEU score for NMT, word error rate for speech recognition and root mean square error for TTS.”
Incorporating Global Visual Features into Attention-based Neural Machine Translation. (2017, methodology): “3), whereas Huang et al. (2016) only use it as a word.”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, methodology): “The Sequential Image Narrative Dataset (SIND) (Huang et al., 2016) is the first large dataset created for image storytelling.”