An empirical study on the effectiveness of images in Multimodal Neural Machine Translation

By Jean-Benoit Delbrouck, Stéphane Dupont. 2017

43 citations recorded by the source corpus.

In state-of-the-art Neural Machine Translation (NMT), an attention mechanism is used during decoding to enhance the translation. At every step, the decoder uses this mechanism to focus on different parts of the source sentence to gather the most useful information before outputting its target word. Recently, the effectiveness of the attention mechanism has also been explored for multi-modal tasks, where it becomes possible to focus both on sentence parts and image regions that they describe. In this paper, we compare several attention mechanism on the multi-modal translation task (English, image → German) and evaluate the ability of the model to make use of images to improve translation. We surpass state-of-the-art scores on the Multi30k data set, we nevertheless identify and report different misbehavior of the machine while translating.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, neutral): “Similar problems were found in VQA [3], where simple baselines without sophisticated multimodal understanding performed remarkably well [94, 35, 1, 26] and in multimodal machine translation [18, 74], where the image was found to matter relatively little [13, 17, 7].”
Visually Grounded Word Embeddings and Richer Visual Features for Improving Multimodal Neural Machine Translation (2017, methodology): “xt can improve translation quality. Meanwhile, few improvements have been made, [9] proposed a doubly-attentive decoder that outperformed all previous baselines with less data and without re-scoring, [10] tried multiple attention models and image attention optimizations such as the gating [3] and pr”
Exploring Deep Multimodal Fusion of Text and Photo for Hate Speech Classification (2019, background): “There is also research initiative in multimodal summarization (Li et al., 2017) and multimodal translation (Calixto et al., 2017; Delbrouck and Dupont, 2017).”
On Vision Features in Multimodal Machine Translation (2022, methodology): “Early attempts mainly focused on enhancing the MMT model by better incorporation of the vision features (Calixto and Liu, 2017; Elliott and Kádár, 2017; Delbrouck and Dupont, 2017).”