All in One: Exploring Unified Video-Language Pre-Training

By Alex Wang, Yixiao Ge, Rui Yan, Yuying Ge, Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, Xiaohu Qie, Mike Zheng Shou. 2022

254 citations recorded by the source corpus.

Mainstream Video-Language Pre-training (VLP) models [10, 26, 64] consist of three parts, a video encoder, a text encoder, and a video-text fusion Transformer. They pursue better performance via utilizing heavier unimodal encoders or multimodal fusion Transformers, resulting in increased parameters with lower efficiency in downstream tasks. In this work, we for the first time introduce an end-to-end VLP model, namely all-in-one Transformer, that embeds raw video and textual signals into joint representations using a unified backbone architecture. We argue that the unique temporal information of video data turns out to be a key barrier hindering the design of a modality-agnostic Transformer. To overcome the challenge, we introduce a novel and effective token rolling operation to encode temporal representations from video clips in a non-parametric manner. The careful design enables the representation learning of both video-text multimodal inputs and unimodal inputs using a unified model. Our pretrained ali-in-one Transformer is transferred to various downstream video-text tasks after fine-tuning, including text-video retrieval, video-question answering, multiple choice and video captioning. State-of-the-art performances with the minimal model FLOPs on ten datasets demonstrate the superiority of our method compared to the competitive counterparts. The code and pretrained models are available at https://github.com/showlab/all-in-one.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

GIT: A Generative Image-to-text Transformer for Vision and Language (2022, background): “[103] Alex Jinpeng Wang, Yixiao Ge, Rui Yan, Yuying Ge, Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou.”
VideoChat: chat-centric video understanding (2023, unknown): “Additionally, prevalent techniques often encompass two or three pretraining tasks, such as masked language modeling [22], video-text matching [43], video-text contrastive learning [49, 46], masked video modeling [41, 44, 46] and video-text masked modeling [12].”
InternVideo: General Video Foundation Models via Generative and Discriminative Learning (2022, methodology): “We mainly follow the practice in [24].”
MovieChat: From Dense Token to Sparse Memory for Long Video Understanding (2023, background): “Such paradigm depends on end-to-end video-language joint training with pretext pre-training tasks such as masked language modeling [47], masked video modeling [80, 87], video-language masked modeling [25], video-text matching [86], and video-text contrastive learning [96].”
InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation (2023, methodology): “Common practices include two or three pretraining tasks, such as masked language modeling [37], video-text matching [38], video-text contrastive learning [23, 30], masked video modeling [35, 36, 30], and video-text masked modeling [39].”
CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Alignment (2022, methodology): “ClipBERT (Lei et al., 2021) 22.0 46.8 59.9 6.0 21.3 49.0 63.5 6.0 VLM (Xu et al., 2021a) 28.1 55.5 67.4 4.0 - - - - MMT (Gabeur et al., 2020) 26.6 57.1 69.6 4.0 28.7 61.4 - 3.3 Support Set (Patrick et al., 2021) 30.1 58.5 69.3 3.0 29.2 61.6 - 3.0 Frozen (Bain et al., 2021) 31.0 59.5 70.5 3.0 28.8 60”
Zero-Shot Video Question Answering via Frozen Bidirectional Language Models (2022, methodology): “[104] Alex Jinpeng Wang, Yixiao Ge, Rui Yan, Yuying Ge, Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou.”
Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning (2023, methodology): “Second, we design a pretraining strategy that effectively leverages cross-modal supervision in the form of transcribed speech from unlabeled narrated videos by reformulating sentence boundaries as pseudo event boundaries, as presented in Section 3.2 and Figure 3.”