Visual Grounding in Video for Unsupervised Word Translation

By Gunnar A. Sigurdsson, Jean-Baptiste Alayrac, Aida Nematzadeh, Lucas Smaira, Mateusz Malinowski, João Carreira, Phil Blunsom, Andrew Zisserman. 2020

52 citations recorded by the source corpus.

There are thousands of actively spoken languages on Earth, but a single visual world. Grounding in this visual world has the potential to bridge the gap between all these languages. Our goal is to use visual grounding to improve unsupervised word mapping between languages. The key idea is to establish a common visual representation between two languages by learning embeddings from unpaired instructional videos narrated in the native language. Given this shared embedding we demonstrate that (i) we can map words between the languages, particularly the 'visual' words; (ii) that the shared embedding provides a good initialization for existing unsupervised text-based word translation techniques, forming the basis for our proposed hybrid visual-text mapping algorithm, MUVE; and (iii) our approach achieves superior performance by addressing the shortcomings of text-based methods -- it is more robust, handles datasets with less commonality, and is applicable to low-resource languages. We apply these methods to translate words from English to French, Korean, and Japanese -- all without any parallel corpora and simply by watching many videos of people speaking while doing things.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Machine Translation

Citations in the corpus

VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation (2021, background): “Future work could consider multilingual VidL datasets [63, 54, 24, 30] as a complementary evaluation to further test systems’ ability on processing information in different languages.”
Broaden Your Views for Self-Supervised Video Learning (2021, unknown): “For the broad views, we consider both visual (RGB frames or optical ﬂow) and audio modalities (spectograms). speech recognition as an additional supervisory signal [2, 3, 53, 55, 56, 72, 74, 79].”
Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training (2022, background): “The cross-modal contrastive loss is shown to be fairly effective in representation learning (Tian et al., 2020; Sigurdsson et al., 2020) and visionlanguage pre-training (Radford et al.”
Visual Attention Methods in Deep Learning: An In-Depth Survey (2022, background): “Prophet Attention: In prophet attention [165], the authors noticed that the conventional attention models are biased and cause deviated focus to the tasks, especially in sequence models such as image captioning [166], [167] and visual grounding [168], [169].”
Source-Free Domain Adaptation for Semantic Segmentation (2021, background): “It is critical for precisely understanding the visual scene and can be applied to numerous potential applications, such as autonomous driving [7], visual grounding [20, 45, 39], and image editing [31].”
Video Understanding as Machine Translation (2020, unknown): “However, as future work we plan to use an internationalized version of the training set [43], which should help assuage such biases.”
UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training (2021, background): “There is also an growing interest in unsupervised multimodal MT [25, 43], where translation between monolingual corpus is augmented via pivoting on image.”