BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers

By Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, Furu Wei. arXiv.org, 2022

439 citations recorded by the source corpus.

Masked image modeling (MIM) has demonstrated impressive results in selfsupervised representation learning by recovering corrupted image patches. However, most existing studies operate on low-level image pixels, which hinders the exploitation of high-level semantics for representation models. In this work, we propose to use a semantic-rich visual tokenizer as the reconstruction target for masked prediction, providing a systematic way to promote MIM from pixel-level to semantic-level. Specifically, we propose vector-quantized knowledge distillation to train the tokenizer, which discretizes a continuous semantic space to compact codes. We then pretrain vision Transformers by predicting the original visual tokens for the masked image patches. Furthermore, we introduce a patch aggregation strategy which associates discrete image patches to enhance global semantic representation. Experiments on image classification and semantic segmentation show that BEIT V2 outperforms all compared MIM methods. On ImageNet-1K (224 size), the base-size BEIT V2 achieves 85.5% top-1 accuracy for fine-tuning and 80.1% top-1 accuracy for linear probing. The large-size BEIT V2 obtains 87.3% top-1 accuracy for ImageNet-1K (224 size) fine-tuning, and 56.7% mIoU on ADE20K for semantic segmentation. The code and pretrained models are available at https://aka.ms/beitv2.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/599be9043ef3571f65758cf36e184c9dc1781baf

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A Cookbook of Self-Supervised Learning (2023, background): “Other works [Bao et al., 2021b, Peng et al., 2022, Wang et al., 2022b] first fine-tune on ImageNet-21k for 60 epochs, and further fine-tune on ImageNet-1k, which represents between 1/5 to 2 times the cost of the pre-training phase.”
GAIA-1: A Generative World Model for Autonomous Driving (2023, methodology): “Inspired by [29], we guide the compression towards meaningful representations by regressing to the latent features of a pre-trained DINO model [30], a self-supervised image model that is known to contain semantic information.”
EVA-02: A Visual Representation for Neon Genesis (2023, methodology): “3-point non-trivial improvement over BEiTv2 [92].”
EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything (2023, methodology): “ViTs [19] have achieved impressive performance in vision applications[5, 20, 26, 34, 39, 44].”
BEATs: Audio Pre-Training with Acoustic Tokenizers (2022, methodology): “Secondly, the discrete label prediction objective can improve the audio modeling efficiency by providing semantic-rich tokens as the pre-training targets and encouraging the model to discard the redundant details, resulting in a superior audio understanding capability with a lower pre-training recou”
Vision Transformer Adapter for Dense Predictions (2022, methodology): “Beneﬁting from multi-modal pre-training (Peng et al., 2022), our ViT-Adapter-L yields 60.9 box AP, which is the best record on COCO test-dev without training on extra detection data such as Objects365 (Shao et al., 2019).”
EVA: Exploring the Limits of Masked Visual Representation Learning at Scale (2022, methodology): “The detailed model configurations are (arch-model size-img resolution-data): ConvNeXt-XL-384px21K [62], SwinV2-L-384px-21K [61], MAE-H-448px-1K [38], DeiT3-L384px-21K [93], EfficientNet-L2&NS-800px-JFT300M [107], BEiTv2-L224px-21K [66], BEiT-L-512px-21K [5], EVA-g-336px-merged30M&21k.”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, background): “Low-Level Targets High-Level Targets Self-Distillation Contrastive / Multi-modal Teacher Algorithm ViT [5] MAE [68] SimMIM [114] Maskfeat [118] BEiT [112] CAE [113] PeCo [119] data2vec [120] SdAE [121] MimCo [122] BEiT v2 [123] Target Raw Pixel HOG VQ-VAE VQ-GAN self MoCo v3 CLIP”
Unleashing Text-to-Image Diffusion Models for Visual Perception (2023, methodology): “For 8K iterations, we find VPDA32 surpass all the baseline methods, including those pre-trained on mask image modelling [17, 38], contrastive learning [7] and supervised learning [27, 29].”
ImageBind One Embedding Space to Bind Them All (2023, methodology): “Feature Alignment Pre-trained CLIP models have been utilized as teachers to supervise other models due to the strength of its visual representations [44, 58, 75].”
DeepNet: Scaling Transformers to 1,000 Layers (2022, methodology): “Speciﬁcally, we encourage D EEP N ET to reconstruct corresponding discrete visual tokens [54], based on the corrupt input images.”
Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks (2022, methodology): “Second, the pretraining task based on masked data modeling has been successfully applied to various modalities, such as texts [DCLT19], images [BDPW22, PDB + 22], and image-text pairs [BWDW22].”
Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks (2022, methodology): “The generalized masked-data-modeling pretraining objective used in U NIFIED -IO is similar to ones used in several recent works (Wang et al., 2022c; Peng et al., 2022; Singh et al., 2022).”
On the Representation Collapse of Sparse Mixture of Experts (2022, methodology): “We will also evaluate the proposed method on vision pretraining (Bao et al., 2022; Peng et al., 2022) and multimodal pretraining (Wang et al., 2022).”