SimMIM: a Simple Framework for Masked Image Modeling

By Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, Han Hu. Computer Vision and Pattern Recognition, 2021

1,859 citations recorded by the source corpus.

This paper presents SimMIM, a simple framework for masked image modeling. We have simplified recently proposed relevant approaches, without the need for special designs, such as block-wise masking and tokenization via discrete VAE or clustering. To investigate what makes a masked image modeling task learn good representations, we systematically study the major components in our framework, and find that the simple designs of each component have revealed very strong representation learning performance: 1) random masking of the input image with a moderately large masked patch size (e.g., 32) makes a powerful pre-text task; 2) predicting RGB values of raw pixels by direct regression performs no worse than the patch classification approaches with complex designs; 3) the prediction head can be as light as a linear layer, with no worse performance than heavier ones. Using ViT-B, our approach achieves 83.8% top-1 fine-tuning accuracy on ImageNet-1K by pre-training also on this dataset, surpassing previous best approach by +0.6%. When applied to a larger model with about 650 million parameters, SwinV2-H, it achieves 87.1% top-1 accuracy on ImageNet-1K using only ImageNet-1K data. We also leverage this approach to address the data-hungry issue faced by large-scale model training, that a 3B model (Swin V2-G) is successfully trained to achieve state-of-the-art accuracy on four representative vision benchmarks using 40× less labelled data than that in previous practice (JFT-3B). The code is available at https://github.com/microsoft/SimMIM.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/9c4753ef43d2928866dc5bf6cec53d03373ec2fa

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

MultiMAE: Multi-modal Multi-task Masked Autoencoders (2022, background): “This is in contrast to the masked autoencoding approaches of SiT [2], BeiT [9] and SimMIM [93], that encode both the masked and visible to-kens.”
ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond (2022, background): “2022), andSimMIM(Xie et al., 2022)The results are summarized in Table 6.”
Revisiting Feature Prediction for Learning Visual Representations from Video (2024, methodology): “…masked image modeling in pixel space by parameterizing the pixel de-coder as a transformer with learnable mask tokens (Doso-vitskiy et al., 2020; Xie et al., 2021; He et al., 2021; Bao et al., 2021), demonstrating a step-change in the representation quality of autoencoding methods (Vincent et…”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, background): “7)–namely, bidirectional encoder representation from image transformers (BEiT) [99], masked AE (MAE) [70], context AE (CAE) [100], and a simple framework for MIM (SimMIM) [101]–have gained signiﬁcant popularity and pose a considerable challenge to the prevailing dominance ofCL.”
RingMo: A Remote Sensing Foundation Model With Masked Image Modeling (2023, methodology): “Masked image modeling (MIM) [31], [32] is a kind of SSL method applying the generative model.”
A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond (2022, methodology): “However, the term masked X modeling, namely masked modeling on X-type data, often refers to the generative case, such as masked language modeling [15], masked image modeling [25], masked point modeling [26].”
MOMENT: A Family of Open Time-series Foundation Models (2024, methodology): “In the zero-shot setting, MOMENT consistently outperformed all statistical interpolation methods with the exception of linear interpolation.”
Visual Prompting via Image Inpainting (2022, background): “Based on the recent success of Vision Transformers (ViTs) [13], multiple works have proposed to hole-ﬁlling a self-supervised pretext task for ViTs [1, 20, 54].”
Masked Autoencoders for Point Cloud Self-supervised Learning (2022, methodology): “These methods have dominated until recent generative SSL methods [17,49,45] result in more competitive performance.”
Dilated Neighborhood Attention Transformer (2022, background): “It also inspired other works that extended it to different tasks beyond the ones explored in the paper, such as generation [48], restoration [22], masked image modeling [45], video action recognition [28], and more.”
Improving Visual Prompt Tuning for Self-supervised Vision Transformers (2023, background): “Masked image modeling (He et al., 2022; Bao et al., 2021; Zhou et al., 2021; Xie et al., 2022), which learns representations by recovering randomly masked patches, is another promising branch of self-supervised learning.”
CoCa: Contrastive Captioners are Image-Text Foundation Models (2022, methodology): “MAE [23] and SimMIM [24] remove the need for an image tokenizer and directly use a light-weight decoder or projection layer to regress pixel values.”
CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformers (2022, methodology): “Although direct mapping is a traditional practice for super-resolution—e.g. SRCNN [4]—it is hardly qualiﬁed as generation; it focuses more on texture transformation.”
A foundation model for clinical-grade computational pathology and rare cancers detection (2024, neutral): “Patch tokens are only matched for a select subset of tokens that are randomly masked out of an input image (for the student), as done in masked image modeling 58 .”
RemoteCLIP: A Vision Language Foundation Model for Remote Sensing (2023, background): “To date, the prevailing approaches are primarily inspired by the success of self-supervised learning (SSL) in computer vision, particularly the masked image modeling (MIM) [3], [10], [11] method.”
Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillation (2022, background): “This usage is so important that solid improvements in this direction can usually attract significant attention, which is right the case for the recent masked image modeling methods [1, 17, 45].”
Masked Autoencoders As Spatiotemporal Learners (2022, background): “Instead of predicting pixels [9, 18, 31, 80], another line of research focuses on the tokenization of the prediction targets [3, 17, 77].”