Video Understanding as Machine Translation

By Bruno Korbar, Fabio Petroni, Rohit Girdhar, Lorenzo Torresani. 2020

29 citations recorded by the source corpus.

With the advent of large-scale multimodal video datasets, especially sequences with audio or transcribed speech, there has been a growing interest in self-supervised learning of video representations. Most prior work formulates the objective as a contrastive metric learning problem between the modalities. To enable effective learning, however, these strategies require a careful selection of positive and negative samples often combined with hand-designed curriculum policies. In this work we remove the need for negative sampling by taking a generative modeling approach that poses the objective as a translation problem between modalities. Such a formulation allows us to tackle a wide variety of downstream video understanding tasks by means of a single unified framework, without the need for large batches of negative samples common in contrastive metric learning. We experiment with the large-scale HowTo100M dataset for training, and report performance gains over the state-of-the-art on several downstream tasks including video classification (EPIC-Kitchens), question answering (TVQA), captioning (TVC, YouCook2, and MSR-VTT), and text-based clip retrieval (YouCook2 and MSR-VTT).

Subfields: Computer Vision and Pattern Recognition, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Machine Translation

Citations in the corpus

VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding (2021, methodology): “CBT (Sun et al., 2019a), HERO (Li et al., 2020b), VideoAsMT (Korbar et al., 2020) and UniVL (Luo et al., 2020) adopt multi-task learning (MTL) to learn retrieval tasks on video and text encoders.”
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval (2021, neutral): “7 VidTranslate [23] ✓ IG65M HowTo100M 136M 14.”
Support-set bottlenecks for video-text representation learning (2020, methodology): “In contrast to Korbar et al. (2020), we show that combining contrastive and generative objectives to pre-train video-text representations on large-scale data such as HowTo100M is very eﬀective.”
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding (2021, background): “CBT (Sun et al., 2019a), HERO (Li et al., 2020b), VideoAsMT (Korbar et al., 2020) and UniVL (Luo et al., 2020) adopt multi-task learning (MTL) for pre-training on retrieval tasks.”
End-to-end Generative Pretraining for Multimodal Video Captioning (2022, methodology): “Works that use multimodal inputs to the encoder, train with masking losses – wherein words or phrases are masked and the objective is to reconstruct the original sentences [23,32] or the masked targets [17] using an autoregressive generator.”
Object-aware Video-language Pre-training for Retrieval (2021, unknown): “0 VidTranslate [16] Arxiv’20 IG65M [136M] HowTo100M 14.”
Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models (2021, methodology): “, 2019) has attracted significant interest in leveraging multimodal pre-training for text→video search (Korbar et al., 2020), captioning (Iashin and Rahtu, 2020), and unsupervised translation via image-based (Surı́s et al.”