VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding

By Hu Xu, Gargi Ghosh, Po-Yao (Bernie) Huang, Prahal Arora, Masoumeh Aminzadeh, Christoph Feichtenhofer, Florian Metze, Luke S. Zettlemoyer. 2021

158 citations recorded by the source corpus.

We present a simplified, task-agnostic multi-modal pre-training approach that can accept either video or text input, or both for a variety of end tasks. Existing pre-training are task-specific by adopting either a single cross-modal encoder that requires both modalities, limiting their use for retrieval-style end tasks or more complex multitask learning with two unimodal encoders, limiting early cross-modal fusion. We instead introduce new pretraining masking schemes that better mix across modalities (e.g. by forcing masks for text to predict the closest video embeddings) while also maintaining separability (e.g. unimodal predictions are sometimes required, without using all the input). Experimental results show strong performance across a wider range of tasks than any previous methods, often outperforming task-specific pre-training. Code is made available at https://github.com/pytorch/fairseq/tree/main/examples/MMPT.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “Others have explored visually-conditioned language generation [17, 114, 119, 124, 132] but have not yet shown good performance in low-data regimes.”
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding (2021, methodology): “, 2018), ActBERT (Zhu and Yang, 2020) and VLM (Xu et al., 2021).”
ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning for Action Recognition (2022, background): “Further, different training methods have also been explored, e.g. , unsupervised learning [74, 23, 84], and video-text contrastive learning [71, 87, 86, 73].”
CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Alignment (2022, unknown): “ClipBERT (Lei et al., 2021) 22.0 46.8 59.9 6.0 21.3 49.0 63.5 6.0 VLM (Xu et al., 2021a) 28.1 55.5 67.4 4.0 - - - - MMT (Gabeur et al., 2020) 26.6 57.1 69.6 4.0 28.7 61.4 - 3.3 Support Set (Patrick et al., 2021) 30.1 58.5 69.3 3.0 29.2 61.6 - 3.0 Frozen (Bain et al., 2021) 31.0 59.5 70.5 3.0 28.8 60”
Deep Multimodal Data Fusion (2024, methodology): “…Image + Text VQA, VCR, ITR UniT [62] 2021 Multi-Transformers Image + Text OD, VQA ClipBERT [90] 2021 Uni-Transformer Video + Text VR VLM [201] 2021 Multi-Transformers Video + Text VQA, VR, VC VATT [3] 2021 Uni-Transformer Video + Text + Audio AC, AEC DeCEMBERT [174] 2021 Uni-Transformer…”
BridgeFormer: Bridging Video-text Retrieval with Multiple Choice Questions (2022, background): “Methods in the second category [20, 21, 23, 38, 40, 44] ensemble texts and videos as inputs to a joint encoder for the cross-modality fusion, followed by a binary classifier which is trained to predict whether videos and texts are aligned or not.”
Egocentric Video-Language Pretraining (2022, background): “With the recent interest boom in computer vision and natural language processing, Video-Language Pretraining (VLP) has prevailed, which aims to learn strong and transferable video-language representation for powering a broad spectrum of video-text downstream tasks, such as video-text retrieval [1, 2”
Multimodal Learning With Transformers: A Survey (2022, background): “architecture design [307], [308], [309] have been introduced, and it is hoped this will spark further investigation.”
MILES: Visual BERT Pre-training with Injected Language Semantics for Video-text Retrieval (2022, background): “Methods in the second category [27,46,55,28,30,50] adopt the “joint-encoder” architectures to interact cross-modality local features through concatenating videos and texts as inputs with a binary classifier to predict whether videos and texts are aligned or not.”