Masked Autoencoders As Spatiotemporal Learners

By Christoph Feichtenhofer, Haoqi Fan, Yanghao Li, Kaiming He. 2022

661 citations recorded by the source corpus.

This paper studies a conceptually simple extension of Masked Autoencoders (MAE) to spatiotemporal representation learning from videos. We randomly mask out spacetime patches in videos and learn an autoencoder to reconstruct them in pixels. Interestingly, we show that our MAE method can learn strong representations with almost no inductive bias on spacetime (only except for patch and positional embeddings), and spacetime-agnostic random masking performs the best. We observe that the optimal masking ratio is as high as 90% (vs. 75% on images), supporting the hypothesis that this ratio is related to information redundancy of the data. A high masking ratio leads to a large speedup, e.g.,>4x in wall-clock time or even more. We report competitive results on several challenging video datasets using vanilla Vision Transformers. We observe that MAE can outperform supervised pre-training by large margins. We further report encouraging results of training on real-world, uncurated Instagram data. Our study suggests that the general framework of masked autoencoding (BERT, MAE, etc.) can be a unified methodology for representation learning with minimal domain knowledge.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Visual Prompt Tuning (2022, background): “The triumph of the Transformer architecture also extends to various computer vision tasks, including image classification [15,39], object detection [8,36], semantic and panoptic segmentation [54,68,60], video understanding [20,61,17] and few-shot learning [14], surpassing previous state-of-the-art a”
A Cookbook of Self-Supervised Learning (2023, methodology): “With ViT for instance, the patch embedding convolutional layer can be transferred from 2D to 3D by repeating the weights along the temporal axis [Feichtenhofer et al., 2022].”
Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles (2023, methodology): “Moreover, MAE pretraining is sparse and can be 4 − 10 × as fast as normal supervised training, making it an already desirable alternative across many domains for more than just accuracy (He et al., 2022; Feichtenhofer et al., 2022; Huang et al., 2022b).”
A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond (2022, result): “Similar investigation has also been investigated in [92] but with spacetime-agnostic random masking.”
VideoMamba: State Space Model for Efficient Video Understanding (2024, positive): “Recently, VideoMAE and ST-MAE [19,75] have showcased the signiﬁcant bene-ﬁts of masked modeling in enhancing a model’s capability for FINE-GRAINED temporal understanding.”
AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition (2022, background): “Recent arts in self-supervised learning [12, 5, 43, 97, 85, 78, 35] can serve as a solution to this challenge.”
InternVideo2: Scaling Foundation Models for Multimodal Video Understanding (2024, background): “…schemes for video representations, including reconstructing videos with masked inputs [He et al., 2022, Tong et al., 2022, Wang et al., 2023b, Feichtenhofer et al., 2022], aligning videos with languages [Li and Wang, 2020, Xu et al., 2021, Yan et al., 2022, Li et al., 2023e], and predicting…”
Token Merging: Your ViT But Faster (2022, methodology): “, 2022) that can put up state-of-the art results while being fast to train; given their lack of assumptions about the data, they can be applied with little or no changes across many modalities (Feichtenhofer et al., 2022; Huang et al., 2022); and they scale well with massive amounts of data (Zhai et”
Real-World Robot Learning with Masked Visual Pre-training (2022, methodology): “Simple and free from dataset or task-speciﬁc augmentations [41], MAE is the state-of-the-art self-supervised framework in computer vision [42, 43, 44, 45], and has been demonstrated to work well for motor control tasks in simulation as well [10].”
Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion (2024, positive): “Masked Autoencoders for images [26] and videos [17] are a popular method for representation learning in pixel space.”
Revisiting Feature Prediction for Learning Visual Representations from Video (2024, methodology): “The use of vision transformers (Dosovitskiy et al., 2020; Li et al., 2022) has become standard practice in self-supervised learning with joint-embedding architectures (Chen et al., 2021; Caron et al., 2021; Oquab et al., 2023; Zhou et al., 2021; Assran et al., 2022), and unlocked masked image…”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, background): “Furthermore,MAE hasbeenextendedtoothermodalitiesbeyondimages[112], [113],[114].”
Should You Mask 15% in Masked Language Modeling? (2022, background): “Recently, a number of works extend MLM training to images and videos and demonstrate strong pre-training results (He et al., 2022; Zhou et al., 2022; Feichtenhofer et al., 2022; Tong et al., 2022) .”
VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking (2023, methodology): “Among them, Transformer [65] with masked autoencoding [16] is becoming a conceptually simple yet effective self-supervised visual learner (e.g., BEiT [3], SimMIM [79], MAE [22] for images, and MaskFeat [76], VideoMAE [63], MAE-ST [18] for videos).”
Transformer-Based Visual Segmentation: A Survey (2023, background): “For video representation learning, most current works [219], [220], [221] verify such representation learning on action or motion learning, such as action recognition.”
EVA: Exploring the Limits of Masked Visual Representation Learning at Scale (2022, background): “EVA processes video data simply via spatial-temporal attention as [33, 92] with no specific architectural adaptation for video related tasks.”
SpectralGPT: Spectral Remote Sensing Foundation Model (2023, methodology): “There have been foundation models designed for video data in computer vision [29], [30], yet there are significant differences between video data and spectral data.”
VideoComposer: Compositional Video Synthesis with Motion Controllability (2023, methodology): “We introduce tube masks [16, 53] to mask out videos and enforce the model to predict the masked regions based on observable information. iv) Sketch sequence.”