VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking

By Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, Yu Qiao. Computer Vision and Pattern Recognition, 2023

754 citations recorded by the source corpus.

Scale is the primary factor for building a powerful foundation model that could well generalize to a variety of downstream tasks. However, it is still challenging to train video foundation models with billions of parameters. This paper shows that video masked autoencoder (VideoMAE) is a scalable and general self-supervised pre-trainer for building video foundation models. We scale the VideoMAE in both model and data with a core design. Specifically, we present a dual masking strategy for efficient pre-training, with an encoder operating on a subset of video tokens and a decoder processing another subset of video tokens. Although VideoMAE is very efficient due to high masking ratio in encoder, masking decoder can still further reduce the overall computational cost. This enables the efficient pre-training of billion-level models in video. We also use a progressive training paradigm that involves an initial pre-training on a diverse multi-sourced unlabeled dataset, followed by a post-pre-training on a mixed labeled dataset. Finally, we successfully train a video ViT model with a billion parameters, which achieves a new state-of-the-art performance on the datasets of Kinetics (90.0% on K400 and 89.9% on K600) and Something-Something (68.7% on V1 and 77.0% on V2). In addition, we extensively verify the pre-trained video ViT models on a variety of downstream tasks, demonstrating its effectiveness as a general video representation learner.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/9fac3d0728a8c833a593446e3e176e90d856df04

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (2023, methodology): “We adopt TSN [75] sampling for all the videos as previous methods [42, 43, 76].”
Revisiting Feature Prediction for Learning Visual Representations from Video (2024, methodology): “Our video pretrained baselines include VideoMAE (Tong et al., 2022), Omni-MAE (Girdhar et al., 2023), Hiera (Ryali et al., 2023), VideoMAEv2 (Wang et al., 2023a), and MVD (Wang et al., 2023b).”
EvalCrafter: Benchmarking and Evaluating Large Video Generation Models (2023, methodology): “In our experiments, we use MMAction2 tool-box [14], specifically the pre-trained VideoMAE V2 [58] model, to infer the human actions in the generated videos.”
InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation (2023, methodology): “Common practices include two or three pretraining tasks, such as masked language modeling [37], video-text matching [38], video-text contrastive learning [23, 30], masked video modeling [35, 36, 30], and video-text masked modeling [39].”
MovieChat: From Dense Token to Sparse Memory for Long Video Understanding (2023, background): “Such paradigm depends on end-to-end video-language joint training with pretext pre-training tasks such as masked language modeling [47], masked video modeling [80, 87], video-language masked modeling [25], video-text matching [86], and video-text contrastive learning [96].”