Context Autoencoder for Self-supervised Representation Learning

By Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, Jingdong Wang. 2022

508 citations recorded by the source corpus.

We present a novel masked image modeling (MIM) approach, context autoencoder (CAE), for self-supervised representation pretraining. We pretrain an encoder by making predictions in the encoded representation space. The pretraining tasks include two tasks: masked representation prediction—predict the representations for the masked patches, and masked patch reconstruction—reconstruct the masked patches. The network is an encoder–regressor–decoder architecture: the encoder takes the visible patches as input; the regressor predicts the representations of the masked patches, which are expected to be aligned with the representations computed from the encoder, using the representations of visible patches and the positions of visible and masked patches; the decoder reconstructs the masked patches from the predicted encoded representations. The CAE design encourages the separation of learning the encoder (representation) from completing the pertaining tasks: masked representation prediction and masked patch reconstruction tasks, and making predictions in the encoded representation space empirically shows the benefit to representation learning. We demonstrate the effectiveness of our CAE through superior transfer performance in downstream tasks: semantic segmentation, object detection and instance segmentation, and classification. The code will be available at https://github.com/Atten4Vis/CAE.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Transformer-Based Visual Segmentation: A Survey (2023, background): “Meanwhile, several works [214] explore the MLP-like architecture for dense prediction.”
A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond (2022, methodology): “Following BEiT [10], CAE [46] ﬁrst trains a image tokenizer via dVAE to generate target visual tokens.”
MaskOCR: Text Recognition with Masked Encoder-Decoder Pretraining (2022, methodology): “We follow the masked image modeling framework, which is recently studied for general image representation pre-training, and adopt the context autoencoder-style method [6] to pre-train the encoder for text image representation learning.”
BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers (2022, background): “…to the reconstruction targets: low-level image elements (e.g., raw pixels; He et al. 2022; Fang et al. 2022; Liu et al. 2022), handcrafted features (e.g., HOG features; Wei et al. 2021), and visual tokens; Bao et al. 2022; Wang et al. 2022; Dong et al. 2021; El-Nouby et al. 2021; Chen et al. 2022.”
Revisiting Feature Prediction for Learning Visual Representations from Video (2024, background): “…et al., 2020), the maturing of masked autoencoding frameworks (Xie et al., 2021; Bao et al., 2021; He et al., 2021), query-based feature pooling (Chen et al., 2022), joint-embedding predictive architectures (JEPA) (LeCun, 2022; Assran et al., 2023; Baevski et al., 2022b), and larger datasets —…”
Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models (2022, background): “[74] Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang.”
HiViT: Hierarchical Vision Transformer Meets Masked Image Modeling (2022, methodology): “Compared to BEiT [1] (we bollow the results from CAE [7]), HiViT-B shows superior results over 7.”
PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers (2021, background): “Current state-of-the-art self-supervised pre-training methods (Dosovitskiy et al. 2020; Bao, Dong, and Wei 2021; He et al. 2021; Xie et al. 2021; Chen et al. 2022; Wei et al. 2021) for vision transformers focus on masked image modeling (MIM), a task of making predictions for masked patches from the ”
Self-Supervised Remote Sensing Feature Learning: Learning Paradigms, Challenges, and Future Works (2022, background): “of generic feature learning [91], [92], [93], [94], [95].”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, background): “7)–namely, bidirectional encoder representation from image transformers (BEiT) [99], masked AE (MAE) [70], context AE (CAE) [100], and a simple framework for MIM (SimMIM) [101]–have gained signiﬁcant popularity and pose a considerable challenge to the prevailing dominance ofCL.”
Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment (2022, methodology): “When fine-tuning the detector on COCO, we find that applying learning rate decay [6, 1, 10, 5] for the components of the detector gives a ∼ 0 .”
Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture (2023, methodology): “Compared to popular methods such as Masked Autoencoders (MAE) [35], Context Autoencoders (CAE) [21], and data2vec [7], which also do not rely on extensive hand-crafted data-augmentations during pretraining, we see that I-JEPA significantly improves linear probing performance, while using less comput”
EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything (2023, methodology): “For the classification task, we compare the performance of ViT back-bones from different pretraining/distillation methods including MAE[26], SSTA[60], DMAE[2], BEiT[3], CAE[12], DINO[6], iBOT[73], DeiT[53], etc.”