Multimodal Attention for Neural Machine Translation

By Ozan Caglayan, Loïc Barrault, Fethi Bougares. 2016

80 citations recorded by the source corpus.

The attention mechanism is an important part of the neural machine translation (NMT) where it was reported to produce richer source representation compared to fixed-length encoding sequence-to-sequence models. Recently, the effectiveness of attention has also been explored in the context of image captioning. In this work, we assess the feasibility of a multimodal attention mechanism that simultaneously focus over an image and its natural language description for generating a description in another language. We train several variants of our proposed attention mechanism on the Multi30k multilingual image captioning dataset. We show that a dedicated attention for each modality achieves up to 1.6 points in BLEU and METEOR compared to a textual NMT baseline.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Machine Translation

Citations in the corpus

VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research (2019, background): “For instance, [27, 23] consider the object features of the images, and [10, 35] import convolutional image features into machine translation.”
Findings of the Third Shared Task on Multimodal Machine Translation (2018, methodology): “LIUMCVC (Task 1) LIUMCVC proposes a reﬁned version of their multimodal attention model (Caglayan et al., 2016), where source-side information from the textual encoder (i.e. last hidden state of the bidirectional gated recurrent units (GRU)) is now used to ﬁl-ter the convolutional feature maps…”
Probing the Need for Visual Context in Multimodal Machine Translation (2019, methodology): “For the MMT model, we explore the basic multimodal attention (DIRECT) (Caglayan et al., 2016) and its hierarchical (HIER) extension (Libovický and Helcl, 2017).”
Evaluating Discourse Phenomena in Neural Machine Translation (2017, methodology): “Inspired by work on multi-modal translation (Caglayan et al., 2016; Huang et al., 2016), multi-encoder translation models have recently been used to incorporate extra-sentential linguistic context in purely textual NMT (Zoph and Knight, 2016; Libovick´y and Helcl, 2017; Wang et al., 2017).”
Emergent Translation in Multi-Agent Communication (2017, methodology): “Additionally, we show In machine translation, our work is related to image-guided approaches (Calixto et al., 2017; Elliott & K ´ ad ´ ar, 2017; Caglayan et al., 2016).”
Imagination Improves Multimodal Translation (2017, methodology): “These features have been used in “double-attention mod-els”, which calculate independent context vectors of the source language hidden states and a convolutional image feature map (Calixto et al., 2016; Caglayan et al., 2016; Calixto et al., 2017a).”
Does Neural Machine Translation Benefit from Larger Context? (2017, background): “…such as multilingual neural translation systems (see, e.g., Firat et al., 2016; Zoph and Knight, 2016), multimodal translation systems (see, e.g., Caglayan et al., 2016; Specia et al., 2016) and syntax-aware neural translation systems (see, e.g., Nadejde et al., 2017; Eriguchi et al., 2016,…”
On Vision Features in Multimodal Machine Translation (2022, background): “Early models of this kind produce a translation given the fused representation of both the visual and textual inputs (Caglayan et al., 2016; Libovický and Helcl, 2017; Calixto and Liu, 2017).”