ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond

By Qiming Zhang, Yufei Xu, Jing Zhang, Dacheng Tao. 2022

305 citations recorded by the source corpus.

Vision transformers have shown great potential in various computer vision tasks owing to their strong capability to model long-range dependency using the self-attention mechanism. Nevertheless, they treat an image as a 1D sequence of visual tokens, lacking an intrinsic inductive bias (IB) in modeling local visual structures and dealing with scale variance, which is instead learned implicitly from large-scale training data with longer training schedules. In this paper, we leverage the two IBs and propose the ViTAE transformer, which utilizes a reduction cell for multi-scale feature and a normal cell for locality. The two kinds of cells are stacked in both isotropic and multi-stage manners to formulate two families of ViTAE models, i.e., the vanilla ViTAE and ViTAEv2. Experiments on the ImageNet dataset as well as downstream tasks on the MS COCO, ADE20K, and AP10K datasets validate the superiority of our models over the baseline and representative models. Besides, we scale up our ViTAE model to 644 M parameters and obtain the state-of-the-art classification performance, i.e., 88.5% Top-1 classification accuracy on ImageNet validation set and the best 91.2% Top-1 classification accuracy on ImageNet Real validation set, without using extra private data. It demonstrates that the introduced inductive bias still helps when the model size becomes large. The source code and pretrained models are publicly available atcode.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

Citations in the corpus

Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learning (2022, background): “As deep learning research advances [34, 41], there is a growing demand for continual learning of neural networks, which requires the network to learn new tasks without forgetting the old knowledge to achieve the stability-plasticity trade-off.”
ISNet: Shape Matters for Infrared Small Target Detection (2022, background): “Deep neural networks, on the contrary, can learn features automatically from a large amount of data covering complex scenes, owing to the end-to-end learning paradigm [13, 21,36,40].”
Advancing Plain Vision Transformer Toward Remote Sensing Foundation Model (2022, methodology): “Network ViT-B [37] ViTAE-B [9] Patch Size 16 16 Embedding Dim 768 768 Head 12 12 Group — 192 Ratio 4 4 Depth 12 12”
Alleviating Semantics Distortion in Unsupervised Low-Level Image-to-Image Translation via Structure Consistency Constraint (2022, background): “Subsequent works improved the performance in more complicated cases such as discrete-continuous mixtures [11, 41], segmentation [66, 70, 72] and continue learning [61, 62].”
Modeling Image Composition for Complex Scene Generation (2022, background): “Recently, transformers not only demonstrate promising results in computer vision tasks [41, 43], but also show potential on conditional visual content generation [4, 21, 26].”
OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Prediction (2023, methodology): “Inspired by recent advances that introduce locality into the transformer [49, 56, 24], we also design the encoder as a hybrid structure.”
Continual Learning with Lifelong Vision Transformer (2022, background): “A popular research direction explores integrating explicit convolution or properties of convolution into the Transformer architecture [16, 73, 86, 92].”
GMFlow: Learning Optical Flow via Global Matching (2021, result): “The top-performing sparse methods (e.g., SuperGlue [34] and LoFTR [38]) adopt Transformers [41] to reason about the mutual relationship between feature descriptors, and the correspondences are extracted with an explicit matching layer (e.g., softmax layer [43]).”
ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation (2022, methodology): “Inspired by the success of ViT [14], many different vision transformer backbones [32, 45, 42, 55, 41, 52, 40, 53] have been proposed, which are typically trained on the ImageNet-1K [13] dataset in a fully supervised setting.”
An Empirical Study of Remote Sensing Pretraining (2022, methodology): “Thus, besides Swin, we also employ another advanced vision transformer named ViTAE [14], [28], where the intrinsic biases of CNN are introduced into the transformer.”
Unifying Flow, Stereo and Depth Estimation (2022, background): “One key reason behind our inferior performance is that RAFT, relying on fully convolutional neural networks, benefits from the inductive biases in convolution layers, which requires a relatively smaller size training data to generalize to a new dataset in comparison with Transformers [96], [97], [98”
Inception Transformer (2022, methodology): “Like previous studies [5, 66], we further fine tune iFormer on the input size of 384× 384, with the weight decay of 1× 10−8, learning rate of 1× 10−5, batch size of 512.”