Auto-scaling Vision Transformers without Training

By Wuyang Chen, Wei Huang, Xianzhi Du, Xiaodan Song, Zhangyang Wang, Denny Zhou. 2022

28 citations recorded by the source corpus.

This work targets automated designing and scaling of Vision Transformers (ViTs). The motivation comes from two pain spots: 1) the lack of efficient and principled methods for designing and scaling ViTs; 2) the tremendous computational cost of training ViT that is much heavier than its convolution counterpart. To tackle these issues, we propose As-ViT, an auto-scaling framework for ViTs without training, which automatically discovers and scales up ViTs in an efficient and principled manner. Specifically, we first design a"seed"ViT topology by leveraging a training-free search process. This extremely fast search is fulfilled by a comprehensive study of ViT's network complexity, yielding a strong Kendall-tau correlation with ground-truth accuracies. Second, starting from the"seed"topology, we automate the scaling rule for ViTs by growing widths/depths to different ViT layers. This results in a series of architectures with different numbers of parameters in a single run. Finally, based on the observation that ViTs can tolerate coarse tokenization in early training stages, we propose a progressive tokenization strategy to train ViTs faster and cheaper. As a unified framework, As-ViT achieves strong performance on classification (83.5% top1 on ImageNet-1k) and detection (52.7% mAP on COCO) without any manual crafting nor scaling of ViT architectures: the end-to-end model design and scaling process cost only 12 hours on one V100 GPU. Our code is available at https://github.com/VITA-Group/AsViT.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Lu Yuan
2. Luc Van Gool
3. Dacheng Tao
4. Jianfeng Gao
5. Bin Xiao
6. Ming-Hsuan Yang
7. Trevor Darrell
8. Xiaogang Wang

In Electrical and Electronic Engineering

1. Lu Yuan
2. Bin Xiao
3. Christopher D. Manning
4. Houwen Peng
5. Chelsea Finn
6. Jianfeng Gao
7. Jianwei Yang
8. Luke Melas-Kyriazi

In Model Architectures

1. Lu Yuan
2. Bin Xiao
3. Mengchen Liu
4. Christopher D. Manning
5. Han Hu
6. Jinnian Zhang
7. Xinyu Liu
8. Kan Wu

In Efficiency & Compression

1. Christopher D. Manning
2. Han Hu
3. Lu Yuan
4. Bin Xiao
5. Xinyu Liu
6. Mengchen Liu
7. Yuqing Yang
8. Houwen Peng

Citations in the corpus

• Fast Model Editing at Scale (2021, background): “Increasingly large models have improved performance on a variety of modern computer vision (Huang et al., 2017; Chen et al., 2022) and especially natural language processing (Vaswani et al.”
• TinyViT: Fast Pretraining Distillation for Small Vision Transformers (2022, background): “…these manually designed or automatically searched small models, our work explores model contraction to generate small models by progressively slimming a large seed model, which can be considered as a complementary work to existing literature on scaling-up large vision transformers [12,78,56,42].”
• EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention (2023, background): “For lightweight mod-Figure els, the configurations of these components need to be care-fully re-designed [7, 8, 39].”
• MaxViT: Multi-Axis Vision Transformer (2022, unknown): “3 975G 74M ◦As-ViT-L [11] 1024×1024 52.”