Self-Supervised MultiModal Versatile Networks

By Jean-Baptiste Alayrac, Adrià Recasens, R. Schneider, Relja Arandjelovi'c, Jason Ramapuram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, Andrew Zisserman. 2020

416 citations recorded by the source corpus.

Videos are a rich source of multi-modal supervision. In this work, we learn representations using self-supervision by leveraging three modalities naturally present in videos: vision, audio and language. To this end, we introduce the notion of a multimodal versatile network -- a network that can ingest multiple modalities and whose representations enable downstream tasks in multiple modalities. In particular, we explore how best to combine the modalities, such that fine-grained representations of audio and vision can be maintained, whilst also integrating text into a common embedding. Driven by versatility, we also introduce a novel process of deflation, so that the networks can be effortlessly applied to the visual data in the form of video or a static image. We demonstrate how such networks trained on large collections of unlabelled video data can be applied on video, video-text, image and audio tasks. Equipped with these representations, we obtain state-of-the-art performance on multiple challenging benchmarks including UCF101, HMDB51 and ESC-50 when compared to previous self-supervised work.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, background): “[2, 3, 38]), but typically across a fixed and predefined set of modalities by means of domain-specific networks.”
Perceiver: General Perception with Iterative Attention (2021, methodology): “Currently, separate feature extractors are used for each modality (Kaiser et al., 2017; Arandjelovic & Zisserman, 2018; Wang et al., 2020b; Chen et al., 2020b; Alayrac et al., 2020; Lee et al., 2020; Xiao et al., 2020) – it is not sensible to concatenate an audio spectrogram or a raw audio sequence ”
Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “Another family of vision-language models is based on contrastive learning [2, 5, 49, 50, 57, 74, 82, 85, 138, 140, 146].”
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, background): “Recent work has explored this paradigm for matching video frames with their transcripts [121], with their audio signal [96, 114], or both [3, 2]; these works likewise perform well on single-modality tasks like audio classification and activity recognition.”
Florence: A New Foundation Model for Computer Vision (2021, neutral): “4 MMV (Alayrac et al., 2020) Video HowTo100M, AudioSet - 31.”
MERLOT: Multimodal Neural Script Knowledge Models (2021, background): “Of late, works have sought to learn multimodal representations transferable to many tasks from uncurated sets of (usually how-to) videos [80, 106, 107, 81, 127, 9, 7, 4]; generally these are applied to video understanding tasks like activity recognition.”
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval (2021, methodology): “A few approaches have been proposed to combat the noise – e.g . us-ing loss functions such as MIL-NCE [42] or using the raw audio [1, 55] directly to increase robustness.”
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text (2021, methodology): “The color augmentation follows [1] and randomizes brightness (max delta = 32/255), saturation (max delta = 0.”
Support-set bottlenecks for video-text representation learning (2020, background): “Other works have leveraged the contrastive loss (Miech et al., 2020) and show that using the raw audio (Alayrac et al., 2020; Rouditchenko et al., 2020) and other modalities (Gabeur et al., 2020) can be used to better align and improve video-text representations.”
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, background): “These models often do not incorporate the audio signal, but even models that do [16–22] still require text captions.”
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding (2021, background): “transfer (Radford et al., 2018, 2019; Brown et al., 2020; Alayrac et al., 2020; Ramesh et al., 2021) that have task-agnostic architecture.”
data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language (2022, background): “…al., 2021) with the aim to produce cross-modal representations which can perform well on multi-modal tasks and with modalities benefiting from each other through joint training (Alayrac et al., 2020; Akbari et al., 2021) with recent methods exploring few-shot learning (Tsimpoukelli et al., 2021).”
Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners (2022, background): “However, video-language pretraining [25, 34, 28, 36, 3, 1, 41] is still hindered by noisy and domain-specific video datasets [22, 37].”
Spatiotemporal Contrastive Video Representation Learning (2020, methodology): “For CVRL pre-trained on K600, it is only worse than the best model of [3, 51, 52, 2] on UCF-101 and outperforms the best model of [3, 52] on HMDB-51, where their pretraining datasets are 108⇥ to 174⇥ larger than K600.”
PandaGPT: One Model To Instruction-Follow Them All (2023, background): “Feature alignment among multiple modalities has attracted great interest for its applications such as cross-modal retrieval [2, 6, 7].”
A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning (2021, methodology): “The strongest previous approaches are using multi-modal input, Vision “V”, Audio “A”, Text “T”, to train a contrastive objective across modalities; XDC [3] performs DeepCluster [8] on (V+A), CVRL [71], GDT [68] and MMV [2] use an objective similar to SimCLR on (V), (V+A), and (V+A+T), with the latte”
VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training (2022, background): “References [1] Jean-Baptiste Alayrac, Adria Recasens, Rosalia Schneider, Relja Arandjelovic, Jason Ramapuram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman.”