iBOT: Image BERT Pre-Training with Online Tokenizer

By Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan L. Yuille, Tao Kong. arXiv.org, 2021

1,143 citations recorded by the source corpus.

The success of language Transformers is primarily attributed to the pretext task of masked language modeling (MLM), where texts are first tokenized into semantically meaningful pieces. In this work, we study masked image modeling (MIM) and indicate the advantages and challenges of using a semantically meaningful visual tokenizer. We present a self-supervised framework iBOT that can perform masked prediction with an online tokenizer. Specifically, we perform self-distillation on masked patch tokens and take the teacher network as the online tokenizer, along with self-distillation on the class token to acquire visual semantics. The online tokenizer is jointly learnable with the MIM objective and dispenses with a multi-stage training pipeline where the tokenizer needs to be pre-trained beforehand. We show the prominence of iBOT by achieving an 82.3% linear probing accuracy and an 87.8% fine-tuning accuracy evaluated on ImageNet-1K. Beyond the state-of-the-art image classification results, we underline emerging local semantic patterns, which helps the models to obtain strong robustness against common corruptions and achieve leading results on dense downstream tasks, eg., object detection, instance segmentation, and semantic segmentation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/9653c070724e44f023e8cc3ec79f0b9e6d59480d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Self-Supervised Learning in Remote Sensing: A review (2022, background): “Since ViT has shown promising results in self-supervised learning [77, 52, 85], and recent advances are exploring the link between large-scale pretraining for both language and vision [58, 86], it deserves more exploration in remote sensing images as well.”
Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling (2021, background): “Training the Transformer only with MPM task leads to an unsatisfactory understanding on high-level semantics of the point clouds, which is also pointed out by the recent work in 2D domain [65].”
Revisiting Feature Prediction for Learning Visual Representations from Video (2024, methodology): “JEPAs have been successfully instantiated in several modalities, such as with audio data (Baevski et al., 2022b) and image data (Zhou et al., 2021; Oquab et al., 2023; Assran et al., 2023).”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, methodology): “Following the introduction of BEiT and MAE, several variants have been proposed. iBOT [98] is an “online tokenizer” adaptation of BEiT, aiming to address the limitation of dVAE in capturing only low-level semantics within local details.”
A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond (2022, background): “To get the merits on both sides, iBOT [79] adopts two losses: a local loss to distill in-view patch to-kens and another global loss to distill between cross-view [CLS] tokens, which makes the target patch tokens more semantically-meaningful [79].”
MultiMAE: Multi-modal Multi-task Masked Autoencoders (2022, methodology): “Vision Transformers are commonly evaluated using the sliding window procedure from MMSegmentation [59] (e.g., [10, 101, 103]).”
AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition (2022, background): “Recent arts in self-supervised learning [12, 5, 43, 97, 85, 78, 35] can serve as a solution to this challenge.”
RoMa: Robust Dense Feature Matching (2023, positive): “…ViT features capture more distinct information than supervised models do, which is demonstrated through label-free self-distillation. iBOT [62] explores MIM within a self-distillation framework to develop a semantically rich visual tokenizer, yielding robust features effective in various…”
Improving Visual Prompt Tuning for Self-supervised Vision Transformers (2023, background): “Masked image modeling (He et al., 2022; Bao et al., 2021; Zhou et al., 2021; Xie et al., 2022), which learns representations by recovering randomly masked patches, is another promising branch of self-supervised learning.”
VC-GPT: Visual Conditioned GPT for End-to-End Generative Vision-and-Language Pre-training (2022, background): “The probability of the token in the t th position is determined by all the past tokens and the visual information.”
Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models (2025, positive): “Previous inﬂuential work like DINOv2 [29], ConvNeXt [25], and EVA [10] demonstrates how incorporating advanced design strategies can revitalize classic methods [16, 48].”
Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs (2024, background): “This category includes contrastive SSL [5, 7, 16, 17] and mask-based SSL [4, 18, 70].”
DeiT III: Revenge of the ViT (2022, positive): “As our approach does not use distillation or multi-crops we have not made a comparison with approaches such as PeCo [11] which use an auxiliary model as a psycho-visual loss and iBoT [63], which uses multi-crop and an exponential moving average of the model.”
EVA: Exploring the Limits of Masked Visual Representation Learning at Scale (2022, background): “Recent work [4,21,30,32,38, 100, 110, 121] (re-)explore pixel / feature regression in MIM, but only in a relatively small model and data scales.”
Masked Siamese Networks for Label-Efficient Learning (2022, methodology): “In contrast to previous work on masked image modelling, the mask-denoising process in MSN is discriminative, rather than generative (He et al., 2021; Xie et al., 2021; Wei et al., 2021; Bao et al., 2021; Zhou et al., 2021).”
Towards Geospatial Foundation Models via Continual Pretraining (2023, methodology): “Masked image modeling (MIM) has been proposed in various forms in recent years, and has recently been found to be particularly effective in the natural image domain, surpassing many contrastive works and being shown to be friendlier to downstream optimization [43, 19, 47, 3, 42] In general, the goal”
data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language (2022, background): “, 2021), during pretraining (Zhou et al., 2021), and others directly predict the input pixels (He et al.”
EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything (2023, methodology): “For the classification task, we compare the performance of ViT back-bones from different pretraining/distillation methods including MAE[26], SSTA[60], DMAE[2], BEiT[3], CAE[12], DINO[6], iBOT[73], DeiT[53], etc.”