DeiT III: Revenge of the ViT

By Hugo Touvron, Matthieu Cord, Herv'e J'egou. 2022

659 citations recorded by the source corpus.

A Vision Transformer (ViT) is a simple neural architecture amenable to serve several computer vision tasks. It has limited built-in architectural priors, in contrast to more recent architectures that incorporate priors either about the input data or of specific tasks. Recent works show that ViTs benefit from self-supervised pre-training, in particular BerT-like pre-training like BeiT. In this paper, we revisit the supervised training of ViTs. Our procedure builds upon and simplifies a recipe introduced for training ResNet-50. It includes a new simple data-augmentation procedure with only 3 augmentations, closer to the practice in self-supervised learning. Our evaluations on Image classification (ImageNet-1k with and without pre-training on ImageNet-21k), transfer learning and semantic segmentation show that our procedure outperforms by a large margin previous fully supervised training recipes for ViT. It also reveals that the performance of our ViT trained with supervision is comparable to that of more recent architectures. Our results could serve as better baselines for recent self-supervised approaches demonstrated on ViT.

Subfields: Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement (2023, methodology): “In computer vision, the state-of-the-art models have upwards of 300 M parameters according to the Timm [63] library (e.g., BEiT [3], DeiT III [60], ConvNeXt [39]) and process inputs This ICCV paper is the Open Access version, provided by the Computer Vision Foundation.”
Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models (2022, background): “4% on ViT-small in [88] trained by advanced layer scale strategy and stronger data augmentation.”
Stabilizing Transformer Training by Preventing Attention Entropy Collapse (2023, methodology): “We compare σReparam, trained for 90 epochs against DeiT3 (Touvron et al., 2022) (trained for 90 [-90E] and 240 [-240E] epochs), an optimized finetuned CLIP (Dong et al.”
Scaling Vision Transformers to 22 Billion Parameters (2023, methodology): “6 mIoU over DeiT-III Large (Touvron et al., 2022) and +2 .”
LocalMamba: Visual State Space Model with Windowed Selective Scan (2024, background): “…the untapped potential of Transformers in visual tasks, sparking a surge of research aimed at refining their architecture design [45] and training methodologies [18, 31, 46, 47, 53], boosting computational efficiency [3,22,33,49], and extending their application scope [8, 25, 26, 38, 44, 56, 58].”
Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles (2023, background): “While architecturally simple, their accuracy (Touvron et al., 2022) and ability to scale (Zhai et al., 2021) make them still a popular choice today.”
UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition (2023, methodology): “…ADE20K semantic segmentation [98], and COCO object detection [44], outperforming the existing large-kernel ConvNets such as RepLKNet [19], SLaK [48], and recent powerful architectures including ConvNeXt V2 [85], FastViT [77], Swin V2 [51] and DeiT III [75] in terms of both accuracy and efﬁciency.”
Vision Transformers Need Registers (2023, methodology): “We produce these plots by running the model on a random subset of ImageNet-22k, and averaging the attention maps for the corresponding tokens at the last layer.”
DINOv2: Learning Robust Visual Features without Supervision (2023, methodology): “To run this sanity check, we apply the finetuning pipeline from Touvron et al. (2022), without tweaking hyper-parameters.”
MixFormer: End-to-End Tracking with Iterative Mixed Attention (2022, methodology): “For MixViT, we carry out the supervised pre-training via the recent DeiT3 [64] model, which successfully trains vanilla ViT models with high performance on the ImageNet-1k and ImageNet-21k.”
InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions (2022, neutral): “9 DeiT III-L‡ [63] T 3842 304M 191G 87.”
EfficientFormer: Vision Transformers at MobileNet Speed (2022, background): “Following this success, several efforts explore the potential of ViT by improving training strategies [3, 4, 5], introducing architecture changes [6, 7], redesigning attention mechanisms [8, 9], and elevating the performance of various vision tasks such as classification [10, 11, 12], segmentation [”