Support-set bottlenecks for video-text representation learning

By Mandela Patrick, Po-Yao (Bernie) Huang, Yuki M. Asano, Florian Metze, Alexander G. Hauptmann, João F. Henriques, Andrea Vedaldi. 2020

271 citations recorded by the source corpus.

The dominant paradigm for learning video-text representations -- noise contrastive learning -- increases the similarity of the representations of pairs of samples that are known to be related, such as text and video from the same sample, and pushes away the representations of all other pairs. We posit that this last behaviour is too strict, enforcing dissimilar representations even for samples that are semantically-related -- for example, visually similar videos or ones that share the same depicted action. In this paper, we propose a novel method that alleviates this by leveraging a generative model to naturally push these related samples together: each sample's caption must be reconstructed as a weighted combination of other support samples' visual representations. This simple idea ensures that representations are not overly-specialized to individual samples, are reusable across the dataset, and results in representations that explicitly encode semantics shared between samples, unlike noise contrastive learning. Our proposed method outperforms others by a large margin on MSR-VTT, VATEX and ActivityNet, for video-to-text and text-to-video retrieval.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, neutral): “Finetuning JEMC (Mithun et al. 2018) 12.5 32.1 42.4 16.0 yes Collaborative Experts (Liu et al. 2019a) 20.6 50.3 64.0 5.3 15.6 40.9 55.2 8.3 yes SSB (Patrick et al. 2020) 28.5 58.6 71.6 3.0 no CLIP4Clip (Luo et al. 2021) 43.1 70.5 81.2 2.0 no ViT-V/32 CLIP2Video (Fang et al. 2021) 43.5 72.3 82.1 2.0…”
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval (2021, methodology): “We also report zero-shot results (Table 4) with no fine-tuning on MSR-VTT, outperforming both MIL-NCE and Support Set that trains on HowTo100M.”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, background): “[85] Mandela Patrick, Po-Yao Huang, Yuki Asano, Florian Metze, Alexander Hauptmann, Joao Henriques, and Andrea Vedaldi.”
VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation (2021, background): “However, existing works [41, 26, 29, 20, 50, 44] in this field are often evaluated on distinct datasets under different experimental settings, making fair comparison difficult between methods.”
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding (2021, methodology): “The result on Youcook2 dataset R@1 ↑ R@5 ↑ R@10 ↑ S UPERVISED HGLMM(Klein et al., 2015) 4.6 14.3 21.6 Coot(Ging et al., 2020) 16.7 40.2 52.3 UniVL (FT-Joint)(Luo et al., 2020) 22.2 52.2 66.2 VideoCLIP (Fine-tuned) 32.2 62.6 75.0 Z ERO - SHOT Random 0.0 0.2 0.3 HowTo100M(Miech et al., 2019) 6.1 17.3 ”
CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval (2021, background): “, 2020), SSB (Patrick et al., 2021), MDMMT (Dzabraev et al.”
Look Before you Speak: Visually Contextualized Utterances (2020, background): “Popular vision and language tasks include visual question answering [4, 5, 22, 28, 58, 61], visual dialog [15, 16, 36, 40, 67], visual captioning [56, 59, 68, 73, 86], visual grounding [18, 22, 57, 83] and video-text retrieval [7, 25, 49, 53, 63].”
Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners (2022, background): “However, video-language pretraining [25, 34, 28, 36, 3, 1, 41] is still hindered by noisy and domain-specific video datasets [22, 37].”
A Metaverse: Taxonomy, Components, Applications, and Open Challenges (2022, methodology): “Multimodal training with videotext representation learning [296].”
Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models (2021, methodology): “, 2020) and inter-modal contrastive objectives (Sigurdsson et al., 2020; Liu et al., 2019; Huang et al., 2019b; Patrick et al., 2021), we employ Transformers to learn contextual multilingual multimodal representations and uniquely models cross-lingual instances.”
HANet: Hierarchical Alignment Networks for Video-Text Retrieval (2021, background): “Video-text retrieval is a non-trivial branch of cross-modal retrieval [3, 13, 15, 21, 27, 31, 36, 40, 46].”
Revisiting the “Video” in Video-Language Understanding (2022, methodology): “We also apply our learnable ATP MSR-VTT DiDeMo ActivityNet R@1 R@5 R@1 R@5 R@1 R@5 Support Set [38] 30. model on standard retrieval benchmarks: DiDeMo [16], MSR-VTT [56], and ActivityNet [23].”