Imagination Improves Multimodal Translation

By Desmond Elliott, Ákos Kádár. 2017

149 citations recorded by the source corpus.

We decompose multimodal translation into two sub-tasks: learning to translate and learning visually grounded representations. In a multitask learning framework, translations are learned in an attention-based encoder-decoder, and grounded representations are learned through image representation prediction. Our approach improves translation performance compared to the state of the art on the Multi30K dataset. Furthermore, it is equally effective if we train the image prediction task on the external MS COCO dataset, and we find improvements if we train the translation model on the external News Commentary parallel text.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Experience Grounds Language (2020, background): “Elliott et al. (2016) takes the classic problem of machine translation and reframes it in the context of visual observations (Elliott and Kádár, 2017).”
Findings of the Third Shared Task on Multimodal Machine Translation (2018, methodology): “The second model is a reimplementation of the Imagination model (Elliott and Kádár, 2017) adapted to the Transformer architecture.”
A Survey of Cross-lingual Word Embedding Models (2017, methodology): “Grounding language in images Similarly to approaches based on word-level aligned comparable data, methods that learn cross-lingual representations using sentence alignment with comparable data do so by associating sentences with images (Elliott & Kádár, 2017).”
On Vision Features in Multimodal Machine Translation (2022, methodology): “Early attempts mainly focused on enhancing the MMT model by better incorporation of the vision features (Calixto and Liu, 2017; Elliott and Kádár, 2017; Delbrouck and Dupont, 2017).”
COCO-CN for Cross-Lingual Image Tagging, Captioning, and Retrieval (2018, methodology): “However, only recently has the topic gained increasing aention [6, 7, 13, 16, 24, 26, 28].”
The MeMAD Submission to the WMT18 Multimodal Translation Task (2018, methodology): “Multi-task learning has also been used, e.g. in the Imagination model (Elliott and Kádár, 2017), where the auxiliary task consists of reconstructing the visual features from the source encoding.”
FigureQA: An Annotated Figure Dataset for Visual Reasoning (2017, background): “The majority of recent methods developed for VQA and related vision-language tasks, such as image captioning [19, 20], video-captioning [21], phrase localization [22], and multi-modal machine translation [23], employ a neural encoder-decoder framework.”
Transferring Knowledge from Vision to Language: How to Achieve it and how to Measure it? (2021, background): “Previous works have evaluated how grounded language representations impact performance on common NLP benchmarks (Sileo, 2021; Kiela et al., 2018; Elliott and Kádár, 2017), but little has been done on investigating grounding specifically as an additional source of knowledge.”
Learning Visually Grounded Sentence Representations (2017, methodology): “Similar ideas were recently successfully applied to multi-modal machine translation (Elliott and Kádár, 2017; Gella et al., 2017; Lee et al., 2017).”
UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training (2021, unknown): “Earlier work introduces vision to an LSTM-based neural MT model via attention to visual context [8, 21], or fusion [6], or multi-task learning [17, 55].”
Visually Grounded Word Embeddings and Richer Visual Features for Improving Multimodal Neural Machine Translation (2017, background): “Recently, [12] introduced a model where visually grounded representations are learned.”
Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation (2021, background): “The Imagination architecture (Elliott and K´ad´ar, 2017; Helcl et al., 2018) decomposes multimodal translation into two sub-tasks: translation task and an auxiliary visual reconstruction task, which encourages the model to learn a visually grounded source sentence representation.”
Unsupervised Multimodal Neural Machine Translation with Pseudo Visual Pivoting (2020, unknown): “Imagination (Elliott and Kádár, 2017) and VAG (Zhou et al., 2018) learns with the auxiliary image reconstruction and source-image-target triplet alignment tasks, respectively.”
Multimodal Transformer for Multimodal Machine Translation (2020, background): “Calixto and Liu (2017) utilize global image features to initialize the encoder/decoder hidden states of RNN. Elliott and K´ad´ar (2017) model the source sentence and reconstruct the image representation jointly via multi-task learning.”