LIUM-CVC Submissions for WMT17 Multimodal Translation Task
By Ozan Caglayan, Walid Aransa, Adrien Bardet, Mercedes García-Martínez, Fethi Bougares, Loïc Barrault, Marc Masana, Luis Herranz, Joost van de Weijer. 2017
71 citations recorded by the source corpus.
This paper describes the monomodal and multimodal Neural Machine Translation systems developed by LIUM and CVC for WMT17 Shared Task on Multimodal Translation. We mainly explored two multimodal architectures where either global visual features or convolutional feature maps are integrated in order to benefit from visual context. Our final systems ranked first for both En-De and En-Fr language pairs according to the automatic evaluation metrics METEOR and BLEU.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Machine Translation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Graham Neubig
- William Chan
- William Yang Wang
- Lucia Specia
- Loïc Barrault
- Luke S. Zettlemoyer
In Computer Vision and Pattern Recognition
- William Yang Wang
- Mohit Bansal
- Desmond Elliott
- Loïc Barrault
- Lucia Specia
- Xin Eric Wang
- Luc Van Gool
- William Chan
In Machine Translation
- Desmond Elliott
- Lucia Specia
- Loïc Barrault
- Ozan Caglayan
- Pranava Swaroop Madhyastha
- Jörg Tiedemann
- Lei Li
- William Chan
Citations in the corpus
- VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research (2019, background): “Additionally, other studies [38, 12, 39, 9] explore the cross-modal feature fusion of images and sentences.”
- Probing the Need for Visual Context in Multimodal Machine Translation (2019, methodology): “All experiments are conducted with nmtpytorch 1 (Caglayan et al., 2017b).”
- Adversarial Evaluation of Multimodal Machine Translation (2018, background): “Consider the three teams that submitted contrastive multimodal and text-only variants of their systems to the 2017 Multimodal Translation Shared Task (Elliott et al., 2017): the University of Le Mans’ multimodal system out-performed their text-only variant (Caglayan et al., Two dogs play with an…”
- Unsupervised Multimodal Neural Machine Translation with Pseudo Visual Pivoting (2020, unknown): “LIUMCVC (Caglayan et al., 2017) uses element-wise multiplication to model the image-text interaction.”
- Illustrative Language Understanding: Large-Scale Visual Grounding with Image Search (2018, methodology): “On the German task, compared to the previously best published results (Caglayan et al., 2017) we do better in BLEU but slightly worse in METEOR.”
- The MeMAD Submission to the WMT18 Multimodal Translation Task (2018, methodology): “We use an affine projection of the image features V ∈ R 80 into a pseudo-word embedding x I ∈ R 512 In the LIUM trg-mul (Caglayan et al., 2017), the target embeddings and visual features are interacted through elementwise multiplication. y ′ j = y j ⊙ tanh ( W decmul · V ) Our initial gating…”