OSU Multimodal Machine Translation System Report

By Mingbo Ma, Dapeng Li, Kai Zhao, Liang Huang. 2017

15 citations recorded by the source corpus.

This paper describes Oregon State University's submissions to the shared WMT'17 task "multimodal translation task I". In this task, all the sentence pairs are image captions in different languages. The key difference between this task and conventional machine translation is that we have corresponding images as additional information for each sentence pair. In this paper, we introduce a simple but effective system which takes an image shared between different languages, feeding it into the both encoding and decoding side. We report our system's performance for English-French and English-German with Flickr30K (in-domain) and MSCOCO (out-of-domain) datasets. Our system achieves the best performance in TER for English-German for MSCOCO dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Machine Translation

Citations in the corpus

VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research (2019, background): “Additionally, other studies [39, 12, 40, 9] explore the cross-modal feature fusion of images and sentences.”
Probing the Need for Visual Context in Multimodal Machine Translation (2019, background): “, 2018) (ii) cross-modal interactions with spatially-unaware global features (Calixto and Liu, 2017; Ma et al., 2017; Caglayan et al., 2017a; Madhyastha et al., 2017) and (iii) the integration of regional features from object detection networks (Huang et al.”
Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description (2017, background): “…of Information and Communications Technology & Nara Institute of Science and Technology (Zhang et al., 2017) OREGONSTATE Oregon State University (Ma et al., 2017) SHEF University of Shefﬁeld (Madhyastha et al., 2017) UvA-TiCC Universiteit van Amsterdam & Tilburg University (Elliott and…”
Adversarial Evaluation of Multimodal Machine Translation (2018, methodology): “2017); the Oregon State University text-only sys-tem outperformed their multimodal variant (Ma et al., 2017); and the performance of the Charles University systems depended on the language pair (Libovick´y and Helcl, 2017).”