ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias

By Yufei Xu, Qiming Zhang, Jing Zhang, Dacheng Tao. 2021

426 citations recorded by the source corpus.

Transformers have shown great potential in various computer vision tasks owing to their strong capability in modeling long-range dependency using the self-attention mechanism. Nevertheless, vision transformers treat an image as 1D sequence of visual tokens, lacking an intrinsic inductive bias (IB) in modeling local visual structures and dealing with scale variance. Alternatively, they require large-scale training data and longer training schedules to learn the IB implicitly. In this paper, we propose a novel Vision Transformer Advanced by Exploring intrinsic IB from convolutions, ie, ViTAE. Technically, ViTAE has several spatial pyramid reduction modules to downsample and embed the input image into tokens with rich multi-scale context by using multiple convolutions with different dilation rates. In this way, it acquires an intrinsic scale invariance IB and is able to learn robust feature representation for objects at various scales. Moreover, in each transformer layer, ViTAE has a convolution block in parallel to the multi-head self-attention module, whose features are fused and fed into the feed-forward network. Consequently, it has the intrinsic locality IB and is able to learn local features and global dependencies collaboratively. Experiments on ImageNet as well as downstream tasks prove the superiority of ViTAE over the baseline transformer and concurrent works. Source code and pretrained models will be available at GitHub.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation (2022, neutral): “Inspired by the success of ViT [14], many different vision transformer backbones [32, 45, 42, 55, 41, 52, 40, 53] have been proposed, which are typically trained on the ImageNet-1K [13] dataset in a fully supervised setting.”
DaViT: Dual Attention Vision Transformers (2022, background): “Hierarchical designs are widely adopted to transformers [37,56,58,16,75,52,40,69,33,1,31,77,48,52,8,30,67,28,64,32,78] in vision.”
Alleviating Semantics Distortion in Unsupervised Low-Level Image-to-Image Translation via Structure Consistency Constraint (2022, background): “Subsequent works improved the performance in more complicated cases such as discrete-continuous mixtures [11, 41], segmentation [66, 70, 72] and continue learning [61, 62].”
ISNet: Shape Matters for Infrared Small Target Detection (2022, background): “Deep neural networks, on the contrary, can learn features automatically from a large amount of data covering complex scenes, owing to the end-to-end learning paradigm [13, 21,36,40].”
OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Prediction (2023, methodology): “Inspired by recent advances that introduce locality into the transformer [49, 56, 24], we also design the encoder as a hybrid structure.”
HiFuse: Hierarchical Multi-Scale Feature Fusion Network for Medical Image Classification (2022, methodology): “The models, such as VitAE, StoHis, TransFuse, CMT, Conformer and so on,Xu et al. [2021], Fu et al. [2022], Zhang et al. [2021], Guo et al. [2022a], Peng et al. [2021] not only inherit the structural advantages of CNN and Transformer, but also verify that the coupling of local features and global…”
DilateFormer: Multi-Scale Dilated Transformer for Visual Recognition (2023, background): “[61], [66], [78], [89], [91] introduce multi-scale information in patch embedding layers [78] or self-attention blocks [42], [66], [91] or add extra branches [10], [61], [89] to perform convolution operation.”
Transformer-Based Visual Segmentation: A Survey (2023, methodology): “The subﬁelds include point cloud segmentation, domain-aware segmentation, label and model efﬁcient segmentation, class agnostic segmentation, tracking, and medical segmentation.”
Visual attention network (2022, background): “, vision transformers) [13], [15], [20], [51], [52], [53], [54], [55], [56], [57], [58], [59], [60], [61], [62], [63], [64] have achieved significantly better performance than the mainstream CNNs on different visual tasks, showing the huge potential of attention-based models.”
Continual Learning with Lifelong Vision Transformer (2022, background): “A popular research direction explores integrating explicit convolution or properties of convolution into the Transformer architecture [16, 73, 86, 92].”
Unifying Flow, Stereo and Depth Estimation (2022, background): “One key reason behind our inferior performance is that RAFT, relying on fully convolutional neural networks, benefits from the inductive biases in convolution layers, which requires a relatively smaller size training data to generalize to a new dataset in comparison with Transformers [96], [97], [98”
ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond (2022, background): “A preliminary version of this work was presented in [85].”
CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers (2022, background): “Recent works have been focused on improving the architectures of attention models, including sparse attention [33, 34, 35, 36, 37, 38, 39], enlarged receptive fields [40, 41], pyramidal designs [42, 43, 44], efficient alternatives [45, 46, 47], etc.”