Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet

By Li Yuan, Yunpeng Chen, Weihao Yu, Yujun Shi, Francis E. H. Tay, Jiashi Feng, Shuicheng Yan. 2021

2,523 citations recorded by the source corpus.

Transformers, which are popular for language modeling, have been explored for solving vision tasks recently, e.g., the Vision Transformer (ViT) for image classification. The ViT model splits each image into a sequence of tokens with fixed length and then applies multiple Transformer layers to model their global relation for classification. However, ViT achieves inferior performance to CNNs when trained from scratch on a midsize dataset like ImageNet. We find it is because: 1) the simple tokenization of input images fails to model the important local structure such as edges and lines among neighboring pixels, leading to low training sample efficiency; 2) the redundant attention backbone design of ViT leads to limited feature richness for fixed computation budgets and limited training samples. To overcome such limitations, we propose a new Tokens-To-Token Vision Transformer (T2T-VTT), which incorporates 1) a layer-wise Tokens-to-Token (T2T) transformation to progressively structurize the image to tokens by recursively aggregating neighboring Tokens into one Token (Tokens-to-Token), such that local structure represented by surrounding tokens can be modeled and tokens length can be reduced; 2) an efficient backbone with a deep-narrow structure for vision transformer motivated by CNN architecture design after empirical study. Notably, T2T-ViT reduces the parameter count and MACs of vanilla ViT by half, while achieving more than 3.0% improvement when trained from scratch on ImageNet. It also outperforms ResNets and achieves comparable performance with MobileNets by directly training on ImageNet. For example, T2T-ViT with comparable size to ResNet50 (21.5M parameters) can achieve 83.3% top1 accuracy in image resolution 384x384 on ImageNet.1

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022, methodology): “[88] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan.”
Early Convolutions Help Transformers See Better (2021, background): “Dominant themes include multi-scale networks [14, 17, 25, 45, 50], increasing depth [42], and locality priors [5, 9, 17, 47, 49].”
VOLO: Vision Outlooker for Visual Recognition (2021, background): “DeiT [51] and T2T-ViT [68] mitigate the problem of ViTs requiring large-scale datasets and propose data efficient ViTs.”
Towards Robust Vision Transformer (2021, methodology): “To answer how is the robustness affected by leveraging low-level features of patches, we compare the original linear projection with two new convolution and tokens-to-tokens embedders, proposed by CeiT and T2T-ViT respectively.”
Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks (2021, methodology): “Transformers have enabled a uniﬁed architecture paradigm for different modalities, which only need the modality-speciﬁc tokenizers to convert inputs from different modalities into uniﬁed input token sequences.”
Scaling Vision Transformers (2021, methodology): “For example, T2T-ViT [50] reduces the number of parameters and compute using a new tokenization and narrower networks.”
Online Continual Learning with Contrastive Vision Transformer (2022, methodology): “Considering the superior modeling capability of Vision Transformers [21] recently demonstrated on computer vision tasks [57,84,26,92,17], we take the utilization of the potential of attention mechanism [74] to develop online CL.”
Meta-attention for ViT-backed Continual Learning (2022, methodology): “In this paper, three popular transformer-based models are used as the backbone: DeiT-Ti [45], DeiT-S [45], T2T-ViT-12 [49].”
CoAtNet: Marrying Convolution and Attention for All Data Sizes (2021, methodology): “In comparison, after the success of ViT and ResNet-ViT [13], another popular line of research starts with a Transformer backbone and tries to incorporate explicit convolution or some desirable properties of convolution into the Transformer backbone [25, 24, 23, 22, 21, 43, 44].”
SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers (2021, methodology): “More recent methods such as T2T ViT [53], CPVT [54], TNT [55], CrossViT [56] and LocalViT [57] introduce tailored changes to ViT to further improve image classiﬁcation performance.”
CvT: Introducing Convolutions to Vision Transformers (2021, methodology): “With fewer paramerters, CvT-13 achieves a 81.6% ImageNet Top-1 accuracy, outperforming PVT-Small [34], T2T-ViT t -14 [41], TNT-S [14] by 1.7%, 0.8%, 0.2% respectively.”
Multiscale Vision Transformers (2021, background): “A separate line of works attempts at modeling visual data with learnt discretized token sequences [110, 89, 17, 115, 21].”
Transformer in Transformer (2021, background): “Nowadays, transformer architectures have been used in a growing number of computer vision tasks [10] such as image recognition [6, 40, 29], object detection [46], and segmentation [43, 38].”
EfficientNetV2: Smaller Models and Faster Training (2021, neutral): “…87M 56B 68 - DeiT-B (ViT+reg) (Touvron et al., 2021) 81.8% 86M 18B 19 - DeiT-B-384 (ViT+reg) (Touvron et al., 2021) 83.1% 86M 56B 68 - T2T-ViT-19 (Yuan et al., 2021) 81.4% 39M 8.4B - - T2T-ViT-24 (Yuan et al., 2021) 82.2% 64M 13B - - ViT-B/16 (21k) (Dosovitskiy et al., 2021) 84.6% 87M 56B 68 -…”