MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound

By Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, Yejin Choi. 2022

261 citations recorded by the source corpus.

As humans, we navigate a multimodal world, building a holistic understanding from all our senses. We introduce @MERLOT RESERVE, a model that represents videos jointly over time - through a new training objective that learns from audio, subtitles, and video frames. Given a video, we replace snippets of text and audio with a MASK token; the model learns by choosing the correct masked-out snippet. Our objective learns faster than alternatives, and performs well at scale: we pretrain on 20 million YouTube videos. Empirical results show that @MERLOT RESERVE learns strong multimodal representations. When finetuned, it sets state-of-the-art on Visual Commonsense Reasoning (VCR), TVQA, and Kinetics-600; outperforming prior work by 5%, 7%, and 1.5% respectively. Ablations show that these tasks benefit from audio pretraining - even VCR, a QA task centered around images (without sound). Moreover, our objective enables out-of-the-box prediction, revealing strong multimodal commonsense understanding. In a fully zero-shot setting, our model obtains competitive results on four video tasks, even outperforming supervised approaches on the recently proposed Situated Reasoning (STAR) benchmark. We analyze why audio enables better vision-language representations, suggesting significant opportunities for future research. We conclude by discussing ethical and societal implications of multimodal pretraining.

Subfields: Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, methodology): “More recent approaches in egocentric perception leverage learned feature representations, utilizing pre-trained convolutional network features (Ryoo, Rothrock, and Matthies 2015), finetuning them (Ma, Fan, and Kitani 2016; Zellers et al. 2022), or training them from scratch (Bambach et al.”
Flamingo: a Visual Language Model for Few-Shot Learning (2022, methodology): “In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models (2023, methodology): “Following the instruction of Transformer [39], pre-trained LLMs demonstrate a powerful text understanding and generation capability [31, 19, 37, 5], and such breakthroughs also benefit the VL modelling [13, 14, 3, 49], where these works add an extra adapter modules [17] in the pre-trained LLMs to al”
Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners (2022, background): “A particular limitation of most recent video-language pretraining frameworks [28, 21, 60, 67, 66, 25, 63] is that they are encoder-only, which means they do not have the ability to generate text from videos for purposes such as captioning [61, 56], question answering [59], and future prediction [23]”
MM-LLMs: Recent Advances in MultiModal Large Language Models (2024, unknown): “…pushing the performance boundaries across a spectrum of downstream tasks (Li et al., 2020; Akbari et al., 2021; Fang et al., 2021; Yan et al., 2021; Li et al., 2021; Radford et al., 2021; Li et al., 2022; Zellers et al., 2022; Zeng et al., 2022b; Yang et al., 2022; Wang et al., 2022a,b).”
Zero-Shot Video Question Answering via Frozen Bidirectional Language Models (2022, unknown): “4 in the main paper, as our model improves over previous work for zero-shot VideoQA [78, 114, 122] by significant margins.”
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action (2023, methodology): “We sample audio and video data from various public datasets including YT-Temporal-1B [215], ACAV100M [105], AudioSet [54], WebVid-10M [13], HD-VILA-10M [200] and Ego4D [60].”
VLP: A Survey on Vision-language Pre-training (2022, unknown): “Some other works, such as AudioCLIP [142] and MERLOT Reserve [143], also shows their unique approaches to learn the cross-modal representations over three modalities.”
Revisiting Feature Prediction for Learning Visual Representations from Video (2024, background): “We believe that further investment towards improving the video pretraining data distribution could lead to substantial gains in downstream image and video tasks.”
V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning (2025, unknown): “The overall dataset includes ego-centric videos from the Something-Something v2 dataset (SSv2) introduced in Goyal et al. (2017), exo-centric action videos from the Kinetics 400, 600, and 700 datasets (Kay et al., 2017; Carreira et al., 2018, 2019), YouTube tutorial videos from HowTo100M (Miech et a”
VideoChat: chat-centric video understanding (2023, unknown): “Although these approaches yield impressive results in multimodal benchmarks, their training relies on limited video-text data, which leads to difficulties in video-only tasks such as action recognition.”
InternVideo: General Video Foundation Models via Generative and Discriminative Learning (2022, methodology): “Starting from the development of image-text pretraining, large-scale video-text pretraining with speciﬁc downstream task ﬁnetuning has become the standard paradigm in the video-language area [26, 30, 32, 46– 51].”
InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation (2023, methodology): “Zellers et al. [4, 5] and Xue et al. [3] proposed YT-Temporal and HD-VILA for Audio-Visual-Language joint learning and high-resolution video crossmodal learning, respectively.”
Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning (2023, methodology): “In contrast, we propose to pretrain on a generic video corpus [121] and show benefits on various domains.”
InternVideo2: Scaling Foundation Models for Multimodal Video Understanding (2024, methodology): “MERLOT Reserve [Zellers et al., 2022] learned video representations using a large-scale dataset of video-speech-transcript pairs.”
See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning (2023, unknown): “Most works [10, 32,37,39,56,68] learn large pre-trained models from massive vision-language data and ﬁnetune them for downstream tasks, which are usually extremely computationally expensive and time-consuming.”
i-Code V2: An Autoregressive Generation Framework over Vision, Language, and Speech Data (2023, background): “…process in many fronts in the era of deep learning, including text summarization (Liu and Lapata, 2019; Yang et al., 2020; Lewis et al., 2020; Zhang et al., 2020; He et al., 2022), dialogue generation (Bao et al., 2020; Zhao et al., 2020; OpenAI, 2022), question answering (Rajpurkar et al.,…”
Objaverse: A Universe of Annotated 3D Objects (2022, methodology): “Language corpora on the web led to large language models like GPT-3 [4]; paired image and text datasets like Conceptual Captions [63] led to vision-and-language pretrained models like VilBERT [42]; YouTube video datasets led to video capable models like Merlot-Reserve [82]; and massive multimodal da”