VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling

By Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Qinghong Lin, William Yang Wang, Lijuan Wang, Zicheng Liu. 2021

255 citations recorded by the source corpus.

A great challenge in video-language (VidL) modeling lies in the disconnection between ﬁxed video representations extracted from image/video understanding models and downstream VidL data. Recent studies try to mitigate this disconnection via end-to-end training. To make it computation-ally feasible, prior works tend to “imagify” video inputs, i.e., a handful of sparsely sampled frames are fed into a 2D CNN, followed by a simple mean-pooling or concatenation to obtain the overall video representations. Although achieving promising results, such simple approaches may lose temporal information that is essential for performing downstream VidL tasks. In this work, we present VIOLET , a fully end-to-end VIdeO-LanguagE Transformer, which adopts a video transformer to explicitly model the temporal dynamics of video inputs. Further, unlike previous studies that found pre-training tasks on video inputs (e.g., masked frame modeling) not very effective, we design a new pretraining task, Masked Visual-token Modeling (MVM), for better video modeling. Speciﬁcally, the original video frame patches are “tokenized” into discrete visual tokens, and the goal is to recover the original visual tokens based on the masked patches. Comprehensive analysis demonstrates the effectiveness of both explicit temporal modeling via video transformer and MVM. As a result, VIOLET achieves new state-of-the-art performance on 5 video question answering tasks and 4 text-to-video retrieval tasks. 1

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Model Architectures

Citations in the corpus

Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks (2021, methodology): “The most widely used objectives are image-text retrieval [2, 5, 43, 55, 63, 73, 74, 75], masked language modeling with image clues [2, 23, 43, 55, 72, 73, 74, 75], and masked region modeling [15, 55, 72, 73, 75].”
Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models (2023, background): “…[39], pre-trained LLMs demonstrate a powerful text understanding and generation capability [31, 19, 37, 5], and such breakthroughs also beneﬁt the VL modelling [13, 14, 3, 49], where these works add an extra adapter modules [17] in the pre-trained LLMs to align visual features to the text space.”
AudioPaLM: A Large Language Model That Can Speak and Listen (2023, neutral): “Such approaches have been applied in audio [Chen et al., 2022c, Bapna et al., 2022, Zhang et al., 2023a] and in vision [Chen et al., 2020, Gan et al., 2020, Fu et al., 2021] as well as combining both audio and video inputs [Shi et al., 2022].”
VideoChat: chat-centric video understanding (2023, unknown): “Additionally, prevalent techniques often encompass two or three pretraining tasks, such as masked language modeling [22], video-text matching [43], video-text contrastive learning [49, 46], masked video modeling [41, 44, 46] and video-text masked modeling [12].”
Zero-Shot Video Question Answering via Frozen Bidirectional Language Models (2022, unknown): “In particular, FrozenBiLM outperforms strong recent baselines such as All-in-one [104] on 2/3 datasets, VIOLET [24] on 3/4 datasets and MERLOT [121] on 4/5 datasets.”
InternVideo2: Scaling Foundation Models for Multimodal Video Understanding (2024, background): “…in building video foundation models (ViFM) include video-text contrastive learning [Li and Wang, 2020, Xu et al., 2021, Wang et al., 2022], masked video modeling [Tong et al., 2022, Wang et al., 2023b, 2022, Fu et al., 2021], and next token prediction [Alayrac et al., 2022, Sun et al., 2023c,a].”
LayoutGPT: Compositional Visual Planning and Generation with Large Language Models (2023, background): “LLMs for Vision Language inputs have been an essential part of many vision language tasks [11, 28, 43, 14, 15].”
Video Graph Transformer for Video Question Answering (2022, background): “More recent works [15,70] collect million-scale user-generated (vastly abundant on the Web) visiontext data [3,51,70] for pretraining, but suffers from huge computational cost to train on such large-scale datasets.”
VIMA: General Robot Manipulation with Multimodal Prompts (2022, background): “In multimodal learning, numerous works (Lu et al., 2022; Wang et al., 2022a; Zellers et al., 2021; 2022; Buch et al., 2022; Fu et al., 2021; Yang et al., 2022) investigate the unification of image, video, audio, and/or language modalities to deliver multi-purpose foundation models, though most of wh”
MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (2023, unknown): “Frame EgoSchema Subset Fullset FrozenBILM [92] 90 - 26.9 VIOLET [18] 5 - 19.9 mPLUG-Owl [ [40].”
InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation (2023, methodology): “Common practices include two or three pretraining tasks, such as masked language modeling [37], video-text matching [38], video-text contrastive learning [23, 30], masked video modeling [35, 36, 30], and video-text masked modeling [39].”
InternVideo: General Video Foundation Models via Generative and Discriminative Learning (2022, methodology): “Starting from the development of image-text pretraining, large-scale video-text pretraining with speciﬁc downstream task ﬁnetuning has become the standard paradigm in the video-language area [26, 30, 32, 46– 51].”
CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Alignment (2022, methodology): “…End-to-end models (Lei et al., 2021; Xue et al., 2022; Zellers et al., 2021; Fu et al., 2021; Huang et al., 2020; 2021b; Xue et al., 2021; Li et al., 2021; Kim et al., 2021; Huang et al., 2021a; Sun et al., 2022) for vision-language pre-training are replacing the traditional approach…”
MovieChat: From Dense Token to Sparse Memory for Long Video Understanding (2023, background): “Such paradigm depends on end-to-end video-language joint training with pretext pre-training tasks such as masked language modeling [47], masked video modeling [80, 87], video-language masked modeling [25], video-text matching [86], and video-text contrastive learning [96].”
Image as a Foreign Language: BEIT Pretraining for Vision and Vision-Language Tasks (2023, methodology): “We follow VIOLET [5] and use the same protocol.”
Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning (2023, methodology): “Second, we design a pretraining strategy that effectively leverages cross-modal supervision in the form of transcribed speech from unlabeled narrated videos by reformulating sentence boundaries as pseudo event boundaries, as presented in Section 3.2 and Figure 3.”