Vision Transformer for Small-Size Datasets

By Seung Hoon Lee, Seunghyun Lee, Byung Cheol Song. arXiv.org, 2021

307 citations recorded by the source corpus.

Recently, the Vision Transformer (ViT), which applied the transformer structure to the image classification task, has outperformed convolutional neural networks. However, the high performance of the ViT results from pre-training using a large-size dataset such as JFT-300M, and its dependence on a large dataset is interpreted as due to low locality inductive bias. This paper proposes Shifted Patch Tokenization (SPT) and Locality Self-Attention (LSA), which effectively solve the lack of locality inductive bias and enable it to learn from scratch even on small-size datasets. Moreover, SPT and LSA are generic and effective add-on modules that are easily applicable to various ViTs. Experimental results show that when both SPT and LSA were applied to the ViTs, the performance improved by an average of 2.96% in Tiny-ImageNet, which is a representative small-size dataset. Especially, Swin Transformer achieved an overwhelming performance improvement of 4.08% thanks to the proposed SPT and LSA.

Subfields: Aerospace Engineering, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/164e41a60120917d13fb69e183ee3c996b6c9414

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Yanyu Li
2. Ju Hu
3. Jian Ren
4. Yang Wen
5. Georgios Evangelidis
6. Kamyar Salahi
7. Yanzhi Wang
8. Sergey Tulyakov

Citations in the corpus

• Rethinking Vision Transformers for MobileNet Size and Speed (2022, background): “With appropriate changes, ViTs demonstrate strong performance in downstream vision tasks [83, 90, 89, 42, 41, 22, 87].”
• EfficientFormer: Vision Transformers at MobileNet Speed (2022, methodology): “Inspired by the competitive performance and global receptive field of transformer models, follow-up works are proposed to refine the architecture [36, 37], explore the relationship between CONV nets and ViT [38, 39, 40], and adapt ViT to different computer vision tasks [13, 41, 42, 43, 44, 45, 46].”