Unsupervised Multimodal Neural Machine Translation with Pseudo Visual Pivoting

By Po-Yao (Bernie) Huang, Junjie Hu, Xiaojun Chang, Alexander G. Hauptmann. 2020

57 citations recorded by the source corpus.

Unsupervised machine translation (MT) has recently achieved impressive results with monolingual corpora only. However, it is still challenging to associate source-target sentences in the latent space. As people speak different languages biologically share similar visual systems, the potential of achieving better alignment through visual content is promising yet under-explored in unsupervised multimodal MT (MMT). In this paper, we investigate how to utilize visual content for disambiguation and promoting latent space alignment in unsupervised MMT. Our model employs multimodal back-translation and features pseudo visual pivoting in which we learn a shared multilingual visual-semantic embedding space and incorporate visually-pivoted captioning as additional weak supervision. The experimental results on the widely used Multi30K dataset show that the proposed model significantly improves over the state-of-the-art methods and generalizes well when images are not available at the testing time.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Machine Translation

Citations in the corpus

Support-set bottlenecks for video-text representation learning (2020, background): “, 2015), autoencoders (Hinton & Salakhutdinov, 2006) and sequence-to-sequence models (Huang et al., 2020; Sutskever et al., 2014) are popular generative models.”
Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models (2021, methodology): “…has attracted significant interest in leveraging multimodal pre-training for text→video search (Korbar et al., 2020), captioning (Iashin and Rahtu, 2020), and unsupervised translation via image-based (Surı́s et al., 2020; Huang et al., 2020b) and video-based (Sigurdsson et al., 2020) alignment.”
IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages (2022, methodology): “, 2021), we have recently witnessed pioneering work on multimodal machine translation (Yao & Wan, 2020; Huang et al., 2020), multilingual visual question answering (Pfeiffer et al.”