All Tokens Matter: Token Labeling for Training Better Vision Transformers

By Zihang Jiang, Qibin Hou, Li Yuan, Daquan Zhou, Yujun Shi, Xiaojie Jin, Anran Wang, Jiashi Feng. 2021

248 citations recorded by the source corpus.

In this paper, we present token labeling -- a new training objective for training high-performance vision transformers (ViTs). Different from the standard training objective of ViTs that computes the classification loss on an additional trainable class token, our proposed one takes advantage of all the image patch tokens to compute the training loss in a dense manner. Specifically, token labeling reformulates the image classification problem into multiple token-level recognition problems and assigns each patch token with an individual location-specific supervision generated by a machine annotator. Experiments show that token labeling can clearly and consistently improve the performance of various ViT models across a wide spectrum. For a vision transformer with 26M learnable parameters serving as an example, with token labeling, the model can achieve 84.4% Top-1 accuracy on ImageNet. The result can be further increased to 86.4% by slightly scaling the model size up to 150M, delivering the minimal-sized model among previous models (250M+) reaching 86%. We also show that token labeling can clearly improve the generalization capability of the pre-trained models on downstream tasks with dense prediction, such as semantic segmentation. Our code and all the training details will be made publicly available at https://github.com/zihangJiang/TokenLabeling.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

VOLO: Vision Outlooker for Visual Recognition (2021, methodology): “2 LV-ViT-M↑384 [32] Transformer 56M 42.”
Transformers in Vision: A Survey (2021, methodology): “Vision Transformer (ViT) [11] is the first work to showcase how Transformers can ‘altogether’ replace standard convolutions in deep neural networks on large-scale computer vision datasets.”
Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations (2022, methodology): “…image classiﬁcation and its follow-ups modify the ViT architecture for not only better visual recognition (Touvron et al., 2021a; Yuan et al., 2021; Zhou et al., 2021) but many other high-level vision tasks, such as object detection (Carion et al., 2020; Zhu et al., 2020; Liu et al., 2021),…”
Neighborhood Attention Transformer (2022, methodology): “We train our model using the commonly used timm package [32], and use the common augmentations and training techniques used in similar works[27, 28, 15, 35, 20, 11, 10], such as CutMix [37], Mixup [38], RandAugment [6], and Random Erasing [40].”
UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning (2022, background): “In contrast, transformer-based backbones are good at processing long sequential data and demonstrate better transfer learning capabilities (Zhou et al., 2021).”
A Survey of Visual Transformers (2021, methodology): “1) Compared with the most structure-improved methods, the basic training strategies, such as DeiT [40] and LV-ViT [43], are more universal for various models.”
DaViT: Dual Attention Vision Transformers (2022, methodology): “Neither token labeling [29] nor distillation [49] is used in all experiments and comparisons.”
Focal Modulation Networks (2022, methodology): “augmentation techniques, such as convolutional FFN [18], token labeling [31] to future work.”
Inception Transformer (2022, background): “LV-ViT [51], LeViT [55] and ViTC [21] further stack several layers of convolutions as the stem for models, which is found helpful in training and achieving better performance.”
OpenEarthMap: A Benchmark Dataset for Global High-Resolution Land Cover Mapping (2022, background): “[21] Zi-Hang Jiang, Qibin Hou, Li Yuan, Daquan Zhou, Yujun Shi, Xiaojie Jin, Anran Wang, and Jiashi Feng.”