Efficient Self-supervised Vision Transformers for Representation Learning

By Chunyuan Li, Jianwei Yang, Pengchuan Zhang, Mei Gao, Bin Xiao, Xiyang Dai, Lu Yuan, Jianfeng Gao. 2021

230 citations recorded by the source corpus.

This paper investigates two techniques for developing efficient self-supervised vision transformers (EsViT) for visual representation learning. First, we show through a comprehensive empirical study that multi-stage architectures with sparse self-attentions can significantly reduce modeling complexity but with a cost of losing the ability to capture fine-grained correspondences between image regions. Second, we propose a new pre-training task of region matching which allows the model to capture fine-grained region dependencies and as a result significantly improves the quality of the learned vision representations. Our results show that combining the two techniques, EsViT achieves 81.3% top-1 on the ImageNet linear probe evaluation, outperforming prior arts with around an order magnitude of higher throughput. When transferring to downstream linear classification tasks, EsViT outperforms its supervised counterpart on 17 out of 18 datasets. The code and models are publicly available: https://github.com/microsoft/esvit

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Model Architectures

In Efficiency & Compression

Citations in the corpus

Unified Contrastive Learning in Image-Text-Label Space (2022, background): “This augmented-view-based paradigm has also been extended to non-contrastive methods [4,7,20,37], where only positive image view pairs are considered in learning.”
Transformers in Vision: A Survey (2021, background): “Efficient self-supervised vision transformer (EsViT) [121] propose a multi-stage design, where neighboring tokens are gradually merged along the hierarchy of the network, and use DINO for self-supervision.”
Focal Self-attention for Local-Global Interactions in Vision Transformers (2021, methodology): “These advancements further inspired the applications of transformer to various vision tasks beyond image classification, such as self-supervised learning [16, 10, 40], object detection [9, 91, 84, 20] and semantic segmentation [61, 65, 86].”
Focal Attention for Long-Range Interactions in Vision Transformers (2021, background): “The results thus inspire researchers to explore the applications of ViT on various vision tasks beyond image classiﬁcation, such as self-supervised learning [14, 9, 39], object detection [8, 85, 79, 18] and semantic segmentation [58, 62, 81].”
iBOT: Image BERT Pre-Training with Online Tokenizer (2021, methodology): “Such practice brings a 0.6% performance gain in terms of linear probing accuracy, which is also observed by a concurrent work, EsViT (Li et al., 2021a).”
Self-Supervised Learning in Remote Sensing: A review (2022, methodology): “Since ViT has shown promising results in SSL [52], [77], [85], and recent advances are exploring the link between large-scale pretraining for both language and vision [58], [86], it deserves more exploration in remote sensing images as well.”
Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillation (2022, methodology): “However, when it comes to vision Transformer backbones [10, 30], the fine-tuning performance becomes inferior to others [43, 26], which may have hindered its appeal for now.”
Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning (2022, methodology): “In viewing patch-based sequence modeling of WSIs in relation to ViTs, we note that the architectural design choice of using Transformer attention enables pretraining of both the tokenization and aggregation layers in ViT models, which is important in preventing MIL models from over- or underfitting ”
A Survey on Vision Transformer (2020, methodology): “Contrastive learning has been applied on vision transformer for unsupervised pretraining [32], [247], [126].”
DINOv2: Learning Robust Visual Features without Supervision (2023, methodology): “First, we run our evaluations for MAE (He et al., 2022), DINO (Caron et al., 2021), SEERv2 (Goyal et al., 2022a), MSN (Assran et al., 2022), EsViT (Li et al., 2022a), Mugs (Zhou et al., 2022b) and iBOT (Zhou et al., 2022a).”
A Cookbook of Self-Supervised Learning (2023, background): “…have been introduced such as the entropy of the singular value distribution [Garrido et al., 2022a], the classical rank estimator [Jing et al., 2022], fitting a power law to the singular value distribution [Ghosh et al., 2022] or the AUC of the singular value distribution [Li et al., 2022a].”