Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description
By Desmond Elliott, Stella Frank, Loïc Barrault, Fethi Bougares, Lucia Specia. 2017
233 citations recorded by the source corpus.
We present the results from the second shared task on multimodal machine translation and multilingual image description. Nine teams submitted 19 systems to two tasks. The multimodal translation task, in which the source sentence is supplemented by an image, was extended with a new language (French) and two new test sets. The multilingual image description task was changed such that at test time, only the image is given. Compared to last year, multimodal systems improved, but text-only systems remain competitive.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Machine Translation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Graham Neubig
- Zhe Gan
- Luke S. Zettlemoyer
- Caiming Xiong
- Percy Liang
- Yejin Choi
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Jianfeng Gao
- Cordelia Schmid
- Andrew Zisserman
- Ming-Hsuan Yang
- Xiaogang Wang
In Multimodal Learning
In Machine Translation
- Desmond Elliott
- Lucia Specia
- Loïc Barrault
- Ozan Caglayan
- Pranava Swaroop Madhyastha
- Ákos Kádár
- Jingjing Liu
- Linjie Li
Citations in the corpus
- MURAL: Multimodal, Multitask Retrieval Across Languages (2021, background): “Elliott et al. (2017) and Barrault et al. (2018) further extend the dataset by collecting human translations of English Flickr30k captions to French and Czech.”
- Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision (2021, methodology): “We test the multilingual model on Multi30k, a multilingual image text retrieval dataset extends Flickr30K (Plum-mer et al., 2015) to German (de) (Elliott et al., 2016), French (fr) (Elliott et al., 2017) and Czech (cs) (Barrault et al., 2018).”
- VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research (2019, background): “Previous studies mainly focus on using images as the visual modality to help machine translation [54, 19, 6].”
- WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning (2021, background): “Consequently there exists only a handful of non-English data collections such as Multi30K-DE (German) [11], DeCOCO (German) [13], Multi30K-FR (French) [10], Multi30K-CS (Czech) [3], YJCaptions26k (Japanese) [23] and MS-COCO-CN (Chinese) [20].”
- Findings of the Third Shared Task on Multimodal Machine Translation (2018, methodology): “We (Elliott et al., 2017)), as described above.”
- Probing the Need for Visual Context in Multimodal Machine Translation (2019, methodology): “Czech and German.”
- Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrieval (2021, methodology): “For multilingual experiments, we use the standard Multi30k dataset [14, 13, 5], which extends Flickr30k with 5 German captions and one French and Czech caption per image.”
- Visually Grounded Neural Syntax Acquisition (2019, unknown): “The hyperparameters H are tuned to maximize: (Young et al., 2014; Elliott et al., 2016, 2017), averaged over 5 runs with different random initialization.”
- (Self-Attentive) Autoencoder-based Universal Language Representation for Machine Translation (2018, background): “Among others, one clear advantage of NMT is that it opens news challenges in MT like multimodal MT (Elliott et al., 2017) or unsupervised MT (Artetxe et al., 2017).”
- Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “The aim of MMT (Specia et al., 2016; Hitschler et al., 2016; Elliott et al., 2017; Barrault et al., 2018) is to translate source language sentences that describe an image into target language.”
- Towards Interlingua Neural Machine Translation (2019, background): “Among others, one clear advantage of NMT is that it opens news challenges in MT like multimodal MT (Elliott et al. 2017).”
- M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training (2020, methodology): “We evaluate M 3 P on Multi30K [28, 29], MSCOCO [16, 30, 31] and MILD.”
- Imagination Improves Multimodal Translation (2017, methodology): “We also evaluate our approach against 16 submissions to the WMT Shared Task on Multimodal Translation and Multilingual Image Description (Elliott et al., 2017).”
- M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training (2021, methodology): “We evaluate M 3 P on multi-lingual image-text retrieval task on two datasets: Multi30K [9, 8] and MSCOCO [2, 25, 20].”
- On Vision Features in Multimodal Machine Translation (2022, methodology): “We reported the results on the Test2016, Test2017 and MSCOCO test sets (Elliott et al., 2017).”
- LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval (2021, methodology): “Multi30K is constructed by manually translating English captions in Flickr30K (Plummer et al., 2015) to German, French, and Czech.”