BEiT: BERT Pre-Training of Image Transformers

By Hangbo Bao, Li Dong, Song-Hao Piao, Furu Wei. 2021

3,776 citations recorded by the source corpus.

We introduce a self-supervised vision representation model BEiT, which stands for Bidirectional Encoder representation from Image Transformers. Following BERT developed in the natural language processing area, we propose a masked image modeling task to pretrain vision Transformers. Specifically, each image has two views in our pre-training, i.e, image patches (such as 16x16 pixels), and visual tokens (i.e., discrete tokens). We first"tokenize"the original image into visual tokens. Then we randomly mask some image patches and fed them into the backbone Transformer. The pre-training objective is to recover the original visual tokens based on the corrupted image patches. After pre-training BEiT, we directly fine-tune the model parameters on downstream tasks by appending task layers upon the pretrained encoder. Experimental results on image classification and semantic segmentation show that our model achieves competitive results with previous pre-training methods. For example, base-size BEiT achieves 83.2% top-1 accuracy on ImageNet-1K, significantly outperforming from-scratch DeiT training (81.8%) with the same setup. Moreover, large-size BEiT obtains 86.3% only using ImageNet-1K, even outperforming ViT-L with supervised pre-training on ImageNet-22K (85.2%). The code and pretrained models are available at https://aka.ms/beit.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Masked Autoencoders Are Scalable Vision Learners (2021, methodology): “Specifically for this variant, we use the DALLE pre-trained dVAE [50] as the tokenizer, following [2].”
Evaluating Large Language Models Trained on Code (2021, background): “, 2018), computer vision (Van Oord et al., 2016; Menick & Kalchbrenner, 2018; Chen et al., 2020; Bao et al., 2021), audio and speech processing (Oord et al.”
Benchmarking Detection Transfer Learning with Vision Transformers (2021, methodology): “BEiT: Since ImageNet-1k pre-trained weights are not available, we use the ofﬁcial BEiT code release [1] to train ViT-B and ViT-L ourselves for 800 epochs (the default training length used in [1]) on unsupervised ImageNet-1k.”
VOLO: Vision Outlooker for Visual Recognition (2021, background): “There are also some works [63], [64], [65] aiming to use self-”
Vector-quantized Image Modeling with Improved VQGAN (2021, methodology): “Similar to our method in predicting image tokens, BEiT (Bao et al., 2021) follows pre-training scheme of BERT Devlin et al. (2019) by learning to recover randomly masked image tokens with a bidirectional transformer.”
SLIP: Self-supervision meets Language-Image Pre-training (2021, methodology): “We note that 100 epochs of training on YFCC15M corresponds to around 1200 epochs of training on ImageNet-1K.”
VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling (2021, methodology): “B. Impact of Masking Strategy To amplify the effectiveness of our MLM and MVM, we introduce Blockwise Masking (BM) and Attended Masking (AM) in Sec.”
VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts (2021, methodology): “Our models adopt the same network conﬁguration as ViT [12] and BE I T [2].”
Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement (2023, background): “In computer vision, the state-of-the-art models have upwards of 300 M parameters according to the Timm [63] library (e.g., BEiT [3], DeiT III [60], ConvNeXt [39]) and process inputs This ICCV paper is the Open Access version, provided by the Computer Vision Foundation.”
GLIGEN: Open-Set Grounded Text-to-Image Generation (2023, background): “In contrast, in the recognition field, the long-standing paradigm has been to build recognition models [29, 37, 76] by starting from a foundation model pretrained on large-scale image data [4,15,16] or image-text pairs [30, 44, 68].”
Visual Prompt Tuning (2022, methodology): “Transformers are also being widely used in recent self-supervised pre-training methods [9,26,2].”
Exploring Plain Vision Transformer Backbones for Object Detection (2022, methodology): “We also use a layer-wise lr decay [9][2] of 0.”
Masked-attention Mask Transformer for Universal Image Segmentation (2021, methodology): “Our best model outperforms the best specialized model, BEiT [2].”
MaskGIT: Masked Generative Image Transformer (2022, neutral): “In vision, the masked modeling in BERT [11] has been extended to image representation learning [2,21] with images quantized to discrete tokens.”
SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model (2023, methodology): “In addition to supervised pre-training, we consider representative self-supervised paradigms that provide pre-trained checkpoints on ViT-B/16, i.e., MoCo v3 [4], BEiT [2] and MAE [13].”
Generative Negative Text Replay for Continual Vision-Language Pretraining (2022, methodology): “Here the pseudo labels are the most confident predictions from the BEiT [4] model pretrained on ImageNet1k.”