Masked Autoencoders Are Scalable Vision Learners

By Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll'ar, Ross B. Girshick. Computer Vision and Pattern Recognition, 2021

12,173 citations recorded by the source corpus.

This paper shows that masked autoencoders (MAE) are scalable self-supervised learners for computer vision. Our MAE approach is simple: we mask random patches of the input image and reconstruct the missing pixels. It is based on two core designs. First, we develop an asymmetric encoder-decoder architecture, with an encoder that operates only on the visible subset of patches (without mask tokens), along with a lightweight decoder that reconstructs the original image from the latent representation and mask tokens. Second, we find that masking a high proportion of the input image, e.g., 75%, yields a nontrivial and meaningful self-supervisory task. Coupling these two designs enables us to train large models efficiently and effectively: we accelerate training (by 3× or more) and improve accuracy. Our scalable approach allows for learning high-capacity models that generalize well: e.g., a vanilla ViT-Huge model achieves the best accuracy (87.8%) among methods that use only ImageNet-1K data. Transfer performance in downstream tasks outperforms supervised pretraining and shows promising scaling behavior.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

View Paper: https://arxiv.org/abs/2111.06377

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Jianfeng Gao
  2. Mohit Bansal
  3. Dacheng Tao
  4. Yoshua Bengio
  5. Quoc V. Le
  6. Sergey Levine
  7. Trevor Darrell
  8. Percy Liang

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Ming-Hsuan Yang
  4. Xiaogang Wang
  5. Trevor Darrell
  6. Jitendra Malik
  7. Yi Yang
  8. Andrew Zisserman

In Representation & Self-Supervised Learning

  1. Ishan Misra
  2. Trevor Darrell
  3. Andrea Vedaldi
  4. Andrew Zisserman
  5. Dacheng Tao
  6. Armand Joulin
  7. Abhinav Kumar Gupta
  8. Yann LeCun

Citations in the corpus