VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training

By Zhan Tong, Yibing Song, Jue Wang, Limin Wang. Neural Information Processing Systems, 2022

2,096 citations recorded by the source corpus.

Pre-training video transformers on extra large-scale datasets is generally required to achieve premier performance on relatively small datasets. In this paper, we show that video masked autoencoders (VideoMAE) are data-efficient learners for self-supervised video pre-training (SSVP). We are inspired by the recent ImageMAE and propose customized video tube masking with an extremely high ratio. This simple design makes video reconstruction a more challenging self-supervision task, thus encouraging extracting more effective video representations during this pre-training process. We obtain three important findings on SSVP: (1) An extremely high proportion of masking ratio (i.e., 90% to 95%) still yields favorable performance of VideoMAE. The temporally redundant video content enables a higher masking ratio than that of images. (2) VideoMAE achieves impressive results on very small datasets (i.e., around 3k-4k videos) without using any extra data. (3) VideoMAE shows that data quality is more important than data quantity for SSVP. Domain shift between pre-training and target datasets is an important issue. Notably, our VideoMAE with the vanilla ViT can achieve 87.4% on Kinetics-400, 75.4% on Something-Something V2, 91.3% on UCF101, and 62.6% on HMDB51, without using any extra data. Code is available at https://github.com/MCG-NJU/VideoMAE.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/4990f7542f0600e0501a7e7a931b32eb7cb804d5

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

EVA: Exploring the Limits of Masked Visual Representation Learning at Scale (2022, methodology): “EVA processes video data simply via spatial-temporal attention as [35, 97] with no specific architectural adaptation for video related tasks.”
Learning from One Continuous Video Stream (2023, methodology): “VideoMAE [43], TubeViT [32], AudioVisualMAE [12], SiameseMAE [15], [2]).”
Should You Mask 15% in Masked Language Modeling? (2022, background): “Recently, a number of works extend MLM training to images and videos and demonstrate strong pre-training results (He et al., 2022; Zhou et al., 2022; Feichtenhofer et al., 2022; Tong et al., 2022) .”
FuXi: a cascade machine learning forecasting system for 15-day global weather forecast (2023, methodology): “To reduce the spatial and temporal dimensions of input and accelerate the training process, the space-time cube-embedding [22] is applied.”
Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles (2023, methodology): “We compare to MViTv2-L (Li et al., 2022c) pretrained with MaskFeat (Wei et al., 2022) and ViT (Dosovitskiy et al., 2021) on video (Feichtenhofer et al., 2022; Tong et al., 2022).”
AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition (2022, methodology): “The experimental settings of image and video mainly follow the ones utilized in MAE [43] and VideoMAE [78], respectively.”
Masked Autoencoders As Spatiotemporal Learners (2022, methodology): “Our work is done independently and concurrently with [66] on a related method.”
Real-World Robot Learning with Masked Visual Pre-training (2022, methodology): “Simple and free from dataset or task-specific augmentations [43], MAE is the state95 of-the-art self-supervised framework in computer vision [44, 45, 46, 47], and has been demonstrated 96 to work well for motor control tasks in simulation [9].”
InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation (2023, methodology): “Common practices include two or three pretraining tasks, such as masked language modeling [37], video-text matching [38], video-text contrastive learning [23, 30], masked video modeling [35, 36, 30], and video-text masked modeling [39].”
Transformer-Based Visual Segmentation: A Survey (2023, background): “For video representation learning, most current works [219], [220], [221] verify such representation learning on action or motion learning, such as action recognition.”
ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning for Action Recognition (2022, background): “, unsupervised learning [67, 20, 76], and video-text contrastive learning [64, 79, 78, 66].”
InternVideo2: Scaling Foundation Models for Multimodal Video Understanding (2024, background): “…similar to other foundational models, InternVideo2 has the potential to embed biases present in its training data and the associated models used during training, such as neural teachers [Tong et al., 2022, Chen et al., 2023c] and language models (LLMs) [Jiang et al., 2023, Zheng et al., 2023].”
MILES: Visual BERT Pre-training with Injected Language Semantics for Video-text Retrieval (2022, background): “When “tube” masking [47] is not adopted and each frame is spatially masked individually, the model achieves the worst results because the model could temporally “interpolate” between frames with visible patches for reconstruction.”
InternVideo: General Video Foundation Models via Generative and Discriminative Learning (2022, methodology): “We follow most rituals from our proposed VideoMAE [23] work to train a vanilla Vision Transformer (ViT) as a video encoder for spatiotemporal modeling, as given in Figure 3 (a).”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, methodology): “Extensions of this work have been explored by Li et al. [225] and Jabri et al. [226].”
A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond (2022, methodology): “Since MAE is found to be more simple yet effective than BEiT, the works in [90], [91] follow the architecture of MAE for simplicity and efficiency.”
A Cookbook of Self-Supervised Learning (2023, background): “More recently, masked auto-encoding objectives for video have been built around the same idea as images, but by masking patches/ tubes of patches in the temporal axis as well [Feichtenhofer et al., 2022, Tong et al., 2022, Girdhar et al., 2022].”
VideoComposer: Compositional Video Synthesis with Motion Controllability (2023, methodology): “We introduce tube masks [16, 53] to mask out videos and enforce the model to predict the masked regions based on observable information. iv) Sketch sequence.”