PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers

By Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, Nenghai Yu. 2021

286 citations recorded by the source corpus.

This paper explores a better prediction target for BERT pre-training of vision transformers. We observe that current prediction targets disagree with human perception judgment. This contradiction motivates us to learn a perceptual prediction target. We argue that perceptually similar images should stay close to each other in the prediction target space. We surprisingly find one simple yet effective idea: enforcing perceptual similarity during the dVAE training. Moreover, we adopt a self-supervised transformer model for deep feature extraction and show that it works well for calculating perceptual similarity. We demonstrate that such learned visual tokens indeed exhibit better semantic meanings, and help pre-training achieve superior transfer performance in various downstream tasks. For example, we achieve 84.5% Top-1 accuracy on ImageNet-1K with ViT-B backbone, outperforming the competitive method BEiT by +1.3% under the same pre-training epochs. Our approach also gets significant improvement on object detection and segmentation on COCO and semantic segmentation on ADE20K. Equipped with a larger backbone ViT-H, we achieve the state-of-the-art ImageNet accuracy (88.3%) among methods using only ImageNet-1K data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

VOLO: Vision Outlooker for Visual Recognition (2021, background): “There are also some works [63], [64], [65] aiming to use self-”
MIC: Masked Image Consistency for Context-Enhanced Domain Adaptation (2022, methodology): “Given a partly masked image, the network is trained to reconstruct properties of the masked areas such as VAE features [2, 16, 50], HOG features [91], or color information [27,96].”
Context Autoencoder for Self-supervised Representation Learning (2022, unknown): “, 2021b), Perceptual Codebook for BEiT (PeCo) (Dong et al., 2021).”
BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers (2022, methodology): “…to the reconstruction targets: low-level image elements (e.g., raw pixels; He et al. 2022; Fang et al. 2022; Liu et al. 2022), handcrafted features (e.g., HOG features; Wei et al. 2021), and visual tokens; Bao et al. 2022; Wang et al. 2022; Dong et al. 2021; El-Nouby et al. 2021; Chen et al. 2022.”
Masked Autoencoders As Spatiotemporal Learners (2022, background): “Instead of predicting pixels [9, 18, 31, 80], another line of research focuses on the tokenization of the prediction targets [3, 17, 77].”
Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, background): “BEiT [69] greatly improves MIM’s performance via masked visual token prediction, and PeCo [70] finds injecting perceptual similarity during visual codebook learning benefits MIM pre-trained representation.”
MILES: Visual BERT Pre-training with Injected Language Semantics for Video-text Retrieval (2022, background): “The follow-up works [54,13] of BEIT confirm this by introducing better image tokenizers that are more aware of the highlevel perceptions.”
A Cookbook of Self-Supervised Learning (2023, background): “The majority of works that followed [Bao et al., 2021b, Zhou et al., 2022a, Dong et al., 2021] focused on this type of evaluation (and sometimes do not report linear/MLP results).”
VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training (2022, background): “[20] Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu.”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, background): “Low-Level Targets High-Level Targets Self-Distillation Contrastive / Multi-modal Teacher Algorithm ViT [5] MAE [68] SimMIM [114] Maskfeat [118] BEiT [112] CAE [113] PeCo [119] data2vec [120] SdAE [121] MimCo [122] BEiT v2 [123] Target Raw Pixel HOG VQ-VAE VQ-GAN self MoCo v3 CLIP”
EVA: Exploring the Limits of Masked Visual Representation Learning at Scale (2022, background): “Recent work [4,21,30,32,38, 100, 110, 121] (re-)explore pixel / feature regression in MIM, but only in a relatively small model and data scales.”
DeiT III: Revenge of the ViT (2022, unknown): “have not made a comparison with approaches such as PeCo [12] which use an auxiliary model as a psycho-visual loss and iBoT [66], which uses multi-crop and an exponential moving average of the model.”
A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond (2022, methodology): “Multiple works [44], [45], [46] have followed this two-stage approach by either improving the tokenizer-based MIM process or seeking an alternative tokenizer.”
BEVT: BERT Pretraining of Video Transformers (2021, methodology): “On K400, we see from [18] instead of the DALL-E tokenizer [46] during pretraining.”