MERLOT: Multimodal Neural Script Knowledge Models

By Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, Yejin Choi. 2021

455 citations recorded by the source corpus.

As humans, we understand events in the visual world contextually, performing multimodal reasoning across time to make inferences about the past, present, and future. We introduce MERLOT, a model that learns multimodal script knowledge by watching millions of YouTube videos with transcribed speech -- in an entirely label-free, self-supervised manner. By pretraining with a mix of both frame-level (spatial) and video-level (temporal) objectives, our model not only learns to match images to temporally corresponding words, but also to contextualize what is happening globally over time. As a result, MERLOT exhibits strong out-of-the-box representations of temporal commonsense, and achieves state-of-the-art performance on 12 different video QA datasets when finetuned. It also transfers well to the world of static images, allowing models to reason about the dynamic context behind visual scenes. On Visual Commonsense Reasoning, MERLOT answers questions correctly with 80.6% accuracy, outperforming state-of-the-art models of similar size by over 3%, even those that make heavy use of auxiliary supervised data (like object bounding boxes). Ablation analyses demonstrate the complementary importance of: 1) training on videos versus static images; 2) scaling the magnitude and diversity of the pretraining video corpus; and 3) using diverse objectives that encourage full-stack multimodal reasoning, from the recognition to cognition level.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Evaluating Large Language Models Trained on Code (2021, background): “…et al., 2020; Bao et al., 2021), audio and speech processing (Oord et al., 2016; 2018; Dhari-wal et al., 2020; Baevski et al., 2020), biology (Alley et al., 2019; Rives et al., 2021), and even across multiple modalities (Das et al., 2017; Lu et al., 2019; Ramesh et al., 2021; Zellers et al., 2021).”
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, background): “Recent works have studied how to ground modern language models, by training them to accept additional environment inputs [23, 24, 25, 26, 27, 28, 29] or to directly output actions [30, 31, 32].”
Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “[142] Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi.”
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, methodology): “We combine our objective with a second contrastive learning approach, tailored to learning visual recognition from scratch: the model must also match each video frame to a contextualized representation of the video’s transcript [126].”
VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling (2021, methodology): “We follow MERLOT [19] to draw colored highlights around the referenced entity (e.g., [PERSON-1] and [CHAIR-2]) in the given image and report performance on the multiple-choice Q→A subtask.”
PaLM-E: An Embodied Multimodal Language Model (2023, background): “…VLMs are capable of simultaneously understanding both images and text, and can be applied to tasks such as visual question answering (Zhou et al., 2020; Zellers et al., 2021b), cap-tioning (Hu et al., 2022), optical character recognition (Li et al., 2021), and object detection (Chen et al., 2021b).”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, background): “8 [127] [122] [30] [120] [5] [3] [5] [69] [92] [100] [100] [103] [125] [78]”
CLiMB: A Continual Learning Benchmark for Vision-and-Language Tasks (2022, positive): “This effect may be due to a shift in the visual domain of VCR: input images have colored boxes drawn on them to represent grounded objects in the question, following previous work [Zellers et al., 2021, Hessel et al., 2022].”
Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion (2021, methodology): “Large scale, pretrained transformer models like BERT [25] provide powerful language representation capacity for EAI [50, 35] and multimodal language and vision [74, 45, 47, 44, 88, 18, 63, 86] tasks.”
PaLI: A Jointly-Scaled Multilingual Language-Image Model (2022, background): “Beside image and language, MERLOT (Zellers et al., 2021) has found success in video understanding and reasoning through video-language pretraining.”
Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners (2022, background): “, MERLOT [67], by a large margin (∼ 4%).”
Zero-Shot Video Question Answering via Frozen Bidirectional Language Models (2022, unknown): “Such data is easy to obtain [6, 74, 121], ensuring the scalability of our approach.”
Multimodal Learning With Transformers: A Survey (2022, background): “emotion recognition [67], multimodal event representation [68], understanding multimodal humor [69], face-body-voice based”
VIMA: General Robot Manipulation with Multimodal Prompts (2022, background): “In multimodal learning, numerous works (Lu et al., 2022; Wang et al., 2022a; Zellers et al., 2021; 2022; Buch et al., 2022; Fu et al., 2021; Yang et al., 2022) investigate the unification of image, video, audio, and/or language modalities to deliver multi-purpose foundation models, though most of wh”
SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning (2021, result): “Recent studies [26, 30, 37–39 , 62] have shown great success on”
CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Alignment (2022, background): “In the past few years, vision-language pre-training has achieved great success on cross-modal representation learning from a large scale of web-crawled data (Radford et al., 2021; Jia et al., 2021; Li et al., 2021; Wang et al., 2021b; Zellers et al., 2021; 2022; Bain et al., 2021).”
ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension (2022, methodology): “Recent hybrid approaches augment CLIP’s architecture with a multi-modal transformer (Li et al., 2021; Zellers et al., 2021).”
Object-aware Video-language Pre-training for Retrieval (2021, background): “The previous works in video-language pretraining focus on aligning the raw-pixel video and raw text with a contrastive loss in both dual-encoder [4,19] and one stream [42] frameworks.”