VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text

By Hassan Akbari, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, Boqing Gong. 2021

756 citations recorded by the source corpus.

We present a framework for learning multimodal representations from unlabeled data using convolution-free Transformer architectures. Specifically, our Video-Audio-Text Transformer (VATT) takes raw signals as inputs and extracts multimodal representations that are rich enough to benefit a variety of downstream tasks. We train VATT end-to-end from scratch using multimodal contrastive losses and evaluate its performance by the downstream tasks of video action recognition, audio event classification, image classification, and text-to-video retrieval. Furthermore, we study a modality-agnostic, single-backbone Transformer by sharing weights among the three modalities. We show that the convolution-free VATT outperforms state-of-the-art ConvNet-based architectures in the downstream tasks. Especially, VATT's vision Transformer achieves the top-1 accuracy of 82.1% on Kinetics-400, 83.6% on Kinetics-600, 72.7% on Kinetics-700, and 41.1% on Moments in Time, new records while avoiding supervised pre-training. Transferring to image classification leads to 78.7% top-1 accuracy on ImageNet compared to 64.7% by training the same Transformer from scratch, showing the generalizability of our model despite the domain gap between videos and images. VATT's audio Transformer also sets a new record on waveform-based audio event recognition by achieving the mAP of 39.4% on AudioSet without any supervised pre-training. VATT's source code is publicly available.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, background): “[2, 3, 38]), but typically across a fixed and predefined set of modalities by means of domain-specific networks.”
All in One: Exploring Unified Video-Language Pre-Training (2022, background): “VATT [1] trains a shared transformer with unimodal inputs to process Video, Audio and Text via multimodal contrastive learning and improves the performance of action recognition.”
Multiview Transformers for Video Recognition (2022, methodology): “Moreover, our model with ImageNet-21K pretraining even outperforms VATT [2], which was pretrained on HowTo100M [47], a dataset consisting of around 100M video clips.”
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, background): “Recent work has explored this paradigm for matching video frames with their transcripts [121], with their audio signal [96, 114], or both [3, 2]; these works likewise perform well on single-modality tasks like audio classification and activity recognition.”
Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks (2021, methodology): “Following this paradigm, recent works [1, 27, 33, 64] adopt Transformers as the backbone for multi-modal applications such as”
UFO: A UniFied TransfOrmer for Vision-Language Representation Learning (2021, background): “The difference with our work is that VATT [2] belongs to the lightfusion, while we focus on unifying the light-fusion and the heavy-fusion.”
MERLOT: Multimodal Neural Script Knowledge Models (2021, background): “Of late, works have sought to learn multimodal representations transferable to many tasks from uncurated sets of (usually how-to) videos [68, 93, 94, 69, 115, 8, 6, 3]; generally these are applied to video understanding tasks like activity recognition.”
data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language (2022, background): “…al., 2021) with the aim to produce cross-modal representations which can perform well on multi-modal tasks and with modalities benefiting from each other through joint training (Alayrac et al., 2020; Akbari et al., 2021) with recent methods exploring few-shot learning (Tsimpoukelli et al., 2021).”
Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners (2022, background): “However, video-language pretraining [25, 34, 28, 36, 3, 1, 41] is still hindered by noisy and domain-specific video datasets [22, 37].”
Human Action Recognition From Various Data Modalities: A Review (2020, neutral): “9 - Multi-Modal GCN [408] 2021RGB,Audio,TextFeature,Score - - - VATT [409] 2021RGB,Audio,Text Feature - - - Doughty et al.”
VLP: A Survey on Vision-language Pre-training (2022, background): “Most previous work on multi-modal pre-training emphasize the joint modeling of language and vision but ignore the information buried in audio[137, 138].”
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge (2022, methodology): “Big dataset such as Common Crawl [24], the Pile [37], LAION [100], YouTube-8M [2] and HowTo100M [77] have been fueling the success of large pre-trained language models [27, 91, 15] and multimodal models [118, 6, 78, 145, 7, 4, 136].”
MultiMAE: Multi-modal Multi-task Masked Autoencoders (2022, methodology): “We use the DPT [66] head to decode layers [3,6,9,12] of the ViT-B encoder into the dense depth map.”
Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts (2022, background): “Some co-train on distinct tasks [34, 35, 36, 22] without aligning or fusing representations—effectively sharing weights across tasks—whereas others include both unimodal aspects and fused multimodal aspects for functionality in both contexts [37].”
MM-LLMs: Recent Advances in MultiModal Large Language Models (2024, neutral): “…significant advancements in recent years, consistently pushing the performance boundaries across a spectrum of downstream tasks (Li et al., 2020; Akbari et al., 2021; Fang et al., 2021; Yan et al., 2021; Li et al., 2021; Radford et al., 2021; Li et al., 2022; Zellers et al., 2022; Zeng et al.,…”