SiT: Self-supervised vIsion Transformer

By Sara Atito Ali Ahmed, Muhammad Awais, Josef Kittler. 2021

166 citations recorded by the source corpus.

Self-supervised learning methods are gaining increasing traction in computer vision due to their recent success in reducing the gap with supervised learning. In natural language processing (NLP) self-supervised learning and transformers are already the methods of choice. The recent literature suggests that the transformers are becoming increasingly popular also in computer vision. So far, the vision transformers have been shown to work well when pretrained either using a large scale supervised data or with some kind of co-supervision, e.g. in terms of teacher network. These supervised pretrained vision transformers achieve very good results in downstream tasks with minimal changes. In this work we investigate the merits of self-supervised learning for pretraining image/vision transformers and then using them for downstream classification tasks. We propose Self-supervised vIsion Transformers (SiT) and discuss several self-supervised training mechanisms to obtain a pretext model. The architectural flexibility of SiT allows us to use it as an autoencoder and work with multiple self-supervised tasks seamlessly. We show that a pretrained SiT can be finetuned for a downstream classification task on small scale datasets, consisting of a few thousand images rather than several millions. The proposed approach is evaluated on standard datasets using common protocols. The results demonstrate the strength of the transformers and their suitability for self-supervised learning. We outperformed existing self-supervised learning methods by large margin. We also observed that SiT is good for few shot learning and also showed that it is learning useful representation by simply training a linear classifier on top of the learned features from SiT. Pretraining, finetuning, and evaluation codes will be available under: https://github.com/Sara-Ahmed/SiT.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Model Architectures

Citations in the corpus

How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers (2021, background): “[1] pre-trains ViT to reconstruct perturbed image patches.”
Context Autoencoder for Self-supervised Representation Learning (2022, unknown): “, 2016), shuffling image patches (Noroozi & Favaro, 2016), denoising pixel-level noise (Atito et al., 2021) and so on.”
A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks (2023, background): “Self-supervised Vision transformer (SiT) allows for the use of the architecture as an autoencoder and seamlessly works with multiple self-supervised tasks (Ahmed et al., 2021).”
Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis (2021, methodology): “For pre-training tasks, (1) masked volume inpainting: the ROI dropping rate is set to 30% (as also used in [2]); the dropped regions are randomly generated and they sum up to reach overall number of voxels; (2) 3D contrastive coding: a feature size of 512 is used as the embedding size; (3) rotation ”
iBOT: Image BERT Pre-Training with Online Tokenizer (2021, methodology): “Predicting masked images parts is a popular self-supervised pretext task drawing on the idea of auto-encoding and has been previously achieved by either recovering raw pixels (Pathak et al., 2016; Atito et al., 2021; Li et al., 2021b) or mask contrastive learning (Henaff, 2020; Zhao et al., 2021).”
Are Large-scale Datasets Necessary for Self-Supervised Pre-training? (2021, unknown): “Chen et al. [35] proposed iGPT, a transformer-based autoregressive model that operates over image pixels, while Atito et al. [36] trained a ViT model on denoising of images where the noise is applied at pixel level.”