Does Multimodality Help Human and Machine for Translation and Image Captioning?

By Ozan Caglayan, Walid Aransa, Yaxing Wang, Marc Masana, Mercedes García-Martínez, Fethi Bougares, Loïc Barrault, Joost van de Weijer. 2016

87 citations recorded by the source corpus.

This paper presents the systems developed by LIUM and CVC for the WMT16 Multimodal Machine Translation challenge. We explored various comparative methods, namely phrase-based systems and attentional recurrent neural networks models trained using monomodal or multimodal data. We also performed a human evaluation in order to estimate the usefulness of multimodal data for human machine translation and image description generation. Our systems obtained the best results for both tasks according to the automatic evaluation metrics BLEU and METEOR.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A Shared Task on Multimodal Machine Translation and Crosslingual Image Description (2016, background): “, 2016) IBM-IITM-Montreal-NYU IBM Research India, IIT Madras, Université de Montréal & New York University LIUM Laboratoire d’Informatique de l’Université du Maine (Caglayan et al., 2016) SHEF University of Sheffield (Shah et al.”
Contextual Parameter Generation for Universal Neural Machine Translation (2018, background): “The same architecture is used in (Caglayan et al., 2016) for translation across multiple modalities.”
Search Engine Guided Non-Parametric Neural Machine Translation (2017, background): “…et al., 2015; Dong et al., 2015), multi-source translation (Zoph and Knight, 2016; Firat et al., 2016b; Garmash and Monz, 2016), multimodal translation (Caglayan et al., 2016) and linguistic knowledge guided translation (Nadejde et al., 2017; Aharoni and Goldberg, 2017; Eriguchi et al., 2017).”
Sheffield MultiMT: Using Object Posterior Predictions for Multimodal Machine Translation (2017, methodology): “Multimodal NMT systems have been introduced (Elliott et al., 2015; Caglayan et al., 2016; Calixto et al., 2016; Huang et al., 2016) to incorporate visual information into NMT approaches, most of which condition the NMT on an image *P. Madhyastha and J.Wang contributed equally to this work.…”
An empirical study on the effectiveness of images in Multimodal Neural Machine Translation (2017, background): “We also compared our results to Caglayan et al. (2016) because our multimodal models are nearly identical with the major exception of the gating scalar (cfr. section 4).”
Using Images to Improve Machine-Translating E-Commerce Product Listings. (2017, methodology): “…(Specia et al., 2016), different research groups have proposed to include images in re-ranking n -best lists generated by an SMT system or directly in a NMT framework with some success (Caglayan et al., 2016; Calixto et al., 2016; Huang et al., 2016; Libovick ´ y et al., 2016; Shah et al., 2016).”
Visually Grounded Word Embeddings and Richer Visual Features for Improving Multimodal Neural Machine Translation (2017, background): “A few attempts [4, 5, 6, 7, 8] have been made during the WMT 2016 Multimodal Machine Translation evaluation campaign.”
Incorporating Global Visual Features into Attention-based Neural Machine Translation. (2017, background): “Nevertheless, researchers have proposed to include global visual features in re-ranking n - best lists generated by a PBSMT system or directly in a purely NMT framework with some success (Caglayan et al., 2016; Calixto et al., 2016; Libovick´y et al., 2016; Shah et al., 2016).”