TinyViT: Fast Pretraining Distillation for Small Vision Transformers

By Kan Wu, Jinnian Zhang, Houwen Peng, Mengchen Liu, Bin Xiao, Jianlong Fu, Lu Yuan. European Conference on Computer Vision, 2022

538 citations recorded by the source corpus.

Vision transformer (ViT) recently has drawn great attention in computer vision due to its remarkable model capability. However, most prevailing ViT models suffer from huge number of parameters, restricting their applicability on devices with limited resources. To alleviate this issue, we propose TinyViT, a new family of tiny and efficient small vision transformers pretrained on large-scale datasets with our proposed fast distillation framework. The central idea is to transfer knowledge from large pretrained models to small ones, while enabling small models to get the dividends of massive pretraining data. More specifically, we apply distillation during pretraining for knowledge transfer. The logits of large teacher models are sparsified and stored in disk in advance to save the memory cost and computation overheads. The tiny student transformers are automatically scaled down from a large pretrained model with computation and parameter constraints. Comprehensive experiments demonstrate the efficacy of TinyViT. It achieves a top-1 accuracy of 84.8% on ImageNet-1k with only 21M parameters, being comparable to Swin-B pretrained on ImageNet-21k while using 4.2 times fewer parameters. Moreover, increasing image resolutions, TinyViT can reach 86.5% accuracy, being slightly better than Swin-L while using only 11% parameters. Last but not the least, we demonstrate a good transfer ability of TinyViT on various downstream tasks. Code and models are available at https://github.com/microsoft/Cream/tree/main/TinyViT.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/2fe71acc2c3f1e75b6149dea72838f0b594ad013

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention (2023, background): “Such large model sizes and the accompanying heavy computational costs make these models unsuitable for applications with real-time requirements [40,78,86].”
EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything (2023, methodology): “This trick has been used in many follow-up works including LeViT[22], EfficientFormer[35], Next-ViT[33], Tiny-ViT[61], Castling-ViT[66], EfficientViT[38].”
Faster Segment Anything: Towards Lightweight SAM for Mobile Applications (2023, methodology): “To make the distillation faster, we follow the practice in Wu et al. [2022] to save the image embeddings beforehand so that we only need to run the forward process once.”
ASAM: Anatomy-Encoded Segment Anything Model for Medical Images (2025, unknown): “ASAM’s three paths are all constructed by TinyViT [21] and initialized with the pre-trained LiteMedSAM [2].”