Masked Autoencoders Are Scalable Vision Learners
By Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll'ar, Ross B. Girshick. Computer Vision and Pattern Recognition, 2021
12,173 citations recorded by the source corpus.
This paper shows that masked autoencoders (MAE) are scalable self-supervised learners for computer vision. Our MAE approach is simple: we mask random patches of the input image and reconstruct the missing pixels. It is based on two core designs. First, we develop an asymmetric encoder-decoder architecture, with an encoder that operates only on the visible subset of patches (without mask tokens), along with a lightweight decoder that reconstructs the original image from the latent representation and mask tokens. Second, we find that masking a high proportion of the input image, e.g., 75%, yields a nontrivial and meaningful self-supervisory task. Coupling these two designs enables us to train large models efficiently and effectively: we accelerate training (by 3× or more) and improve accuracy. Our scalable approach allows for learning high-capacity models that generalize well: e.g., a vanilla ViT-Huge model achieves the best accuracy (87.8%) among methods that use only ImageNet-1K data. Transfer performance in downstream tasks outperforms supervised pretraining and shows promising scaling behavior.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning
View Paper: https://arxiv.org/abs/2111.06377
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Mohit Bansal
- Dacheng Tao
- Yoshua Bengio
- Quoc V. Le
- Sergey Levine
- Trevor Darrell
- Percy Liang
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Xiaogang Wang
- Trevor Darrell
- Jitendra Malik
- Yi Yang
- Andrew Zisserman
In Representation & Self-Supervised Learning
- Ishan Misra
- Trevor Darrell
- Andrea Vedaldi
- Andrew Zisserman
- Dacheng Tao
- Armand Joulin
- Abhinav Kumar Gupta
- Yann LeCun
Citations in the corpus
- Benchmarking Detection Transfer Learning with Vision Transformers (2021, methodology): “We use the DeiT released weights [36] for ViT-B and the ViT-L weights from [16], which uses an even stronger training formula than DeiT to avoid overfitting (moreover, the DeiT release does not include ViT-L).”
- VOLO: Vision Outlooker for Visual Recognition (2021, background): “There are also some works [63], [64], [65] aiming to use self-”
- SLIP: Self-supervision meets Language-Image Pre-training (2021, methodology): “Recent methods for self-supervised learning also propose a variety of alternatives to the contrastive objective such as self-distillation [25, 6], or input reconstruction [1, 26].”
- Segment Anything (2023, methodology): “Motivated by scalability and powerful pre-training methods, we use an MAE [47] pre-trained Vision Transformer (ViT) [33] minimally adapted to process high resolution inputs [62].”
- GLIGEN: Open-Set Grounded Text-to-Image Generation (2023, background): “In contrast, in the recognition field, the long-standing paradigm has been to build recognition models [29, 37, 76] by starting from a foundation model pretrained on large-scale image data [4,15,16] or image-text pairs [30, 44, 68].”
- ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models (2022, neutral): “For IC, we also compare CLIP against other language-free visual models including MoCov3, MAE, ViT, DeiT in Appendix.”
- Visual Prompt Tuning (2022, methodology): “Backbone Pre-trained Objective Pre-trained Dataset # params (M) Feature dim d Batch Size Pre-trained Model ViT-B/16 [16] Supervised ImageNet-21k 85 768 2048 / 1280 / 128 / 128 / 64 checkpoint ViT-L/16 [16] 307 1024 2048 / 640 / 64 / 64 / 32 checkpoint ViT-H/14 [16] 630 1280 1024 / 240 / 28 / 28 / 14”
- Multiview Compressive Coding for 3D Reconstruction (2023, methodology): “Our decoder is a lighter-weight 8-layer 512-dimensional transformer, following MAE [20].”
- SimpleClick: Interactive Image Segmentation with Simple Vision Transformers (2022, methodology): “MAE [21]) to these tasks.”
- Exploring Plain Vision Transformer Backbones for Object Detection (2022, methodology): “With Masked Autoencoder (MAE) [23] pretraining, our plain-backbone detector can outperform the hierarchical counterparts that are pre-trained on ImageNet-1K/21K [11] with supervision (Figure 3).”
- Occluded Video Instance Segmentation: A Benchmark (2021, methodology): “With the development of self-supervised learning (He et al., 2021), exploiting the unlimited amounts of unlabeled data for model pre-training, then transferring the pre-trained model into OVIS will largely enhance the discriminative power of frame embeddings.”
- MaskGIT: Masked Generative Image Transformer (2022, neutral): “In vision, the masked modeling in BERT [11] has been extended to image representation learning [2,21] with images quantized to discrete tokens.”
- SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model (2023, methodology): “1 The reported performance of sequential fine-tuning (Seq FT) is improved by more than DeiT [35] is a strong supervised method for (pre-)training vision transformer, while MoCo v3 [4], MAE [13] and BEiT [2] are representative self-supervised methods.”
- Towards Label-Efficient Incremental Learning: A Survey (2023, background): “However, recent studies like MAE show promise of reconstruction-based objectives over contrastive-based counterparts, which we believe is worthy of exploration [He et al. , 2022].”
- CoCa: Contrastive Captioners are Image-Text Foundation Models (2022, methodology): “As also discussed in [23], linear-evaluation struggles to accurately measure learned representations and we find the attentional poolers are more practical for real-world applications.”
- A ConvNet for the 2020s (2022, methodology): “We use the supervised training results from DeiT [73] for ViT-S/B and MAE [26] for ViT-L, as they employ improved training procedures over the original ViTs [20].”