VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learning

By Hao Hao Tan, Jie Lei, Thomas Wolf, Mohit Bansal. 2021

77 citations recorded by the source corpus.

Video understanding relies on perceiving the global content and modeling its internal connections (e.g., causality, movement, and spatio-temporal correspondence). To learn these interactions, we apply a mask-then-predict pre-training task on discretized video tokens generated via VQ-VAE. Unlike language, where the text tokens are more independent, neighboring video tokens typically have strong correlations (e.g., consecutive video frames usually look very similar), and hence uniformly masking individual tokens will make the task too trivial to learn useful representations. To deal with this issue, we propose a block-wise masking strategy where we mask neighboring video tokens in both spatial and temporal domains. We also add an augmentation-free contrastive learning method to further capture the global content by predicting whether the video clips are sampled from the same video. We pre-train our model on uncurated videos and show that our pre-trained model can reach state-of-the-art results on several video understanding datasets (e.g., SSV2, Diving48). Lastly, we provide detailed analyses on model scalability and pre-training method design. Code is released at https://github.com/airsplay/vimpac.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling (2021, methodology): “VIMPAC [71] takes a step further to completely remove the raw pixel inputs from the training procedure.”
A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond (2022, methodology): “VIMPAC [89] proposes a different single-stage masked video modeling method, which includes a block-wise masking strategy for videos and augmentation-free contrastive learning loss to learn the global features.”
iBOT: Image BERT Pre-Training with Online Tokenizer (2021, methodology): “Recently, it is formulated into MIM (Bao et al., 2021; Tan et al., 2021) with a discrete VAE (Rolfe, 2017; Ramesh et al., 2021) as visual tokenizer.”
VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking (2023, background): “BEiT [3], BEVT [72], and VIMPAC [60] learned visual representations by predicting discrete tokens.”
Masked Feature Prediction for Self-Supervised Visual Pre-Training (2021, methodology): “BEiT [2] and VIMPAC [82] encode masked patches with discrete variational autoencoder (dVAE) [73, 87].”
Masked Autoencoders As Spatiotemporal Learners (2022, background): “Previous works focus on tokenizing the prediction targets for the use of videos [65, 73, 77].”