Co-Scale Conv-Attentional Image Transformers

By Weijian Xu, Yifan Xu, Tyler A. Chang, Zhuowen Tu. 2021

466 citations recorded by the source corpus.

In this paper, we present Co-scale conv-attentional image Transformers (CoaT), a Transformer-based image classifier equipped with co-scale and conv-attentional mechanisms. First, the co-scale mechanism maintains the integrity of Transformers’ encoder branches at individual scales, while allowing representations learned at different scales to effectively communicate with each other; we design a series of serial and parallel blocks to realize the co-scale mechanism. Second, we devise a conv-attentional mechanism by realizing a relative position embedding formulation in the factorized attention module with an efficient convolution-like implementation. CoaT empowers image Transformers with enriched multi-scale and contextual modeling capabilities. On ImageNet, relatively small CoaT models attain superior classification results compared with similar-sized convolutional neural networks and image/vision Transformers. The effectiveness of CoaT’s backbone is also illustrated on object detection and instance segmentation, demonstrating its applicability to downstream computer vision tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Online Continual Learning with Contrastive Vision Transformer (2022, methodology): “Besides, LeViT, CoAT, and CCT contain CNN structures to obtain inductive biases, which still suffer from catastrophic forgetting in online CL.”
Transformers in Vision: A Survey (2021, methodology): “CoaT [226], is a hybrid hierarchical pyramid design, with serial and parallel blocks, where the serial block is similar to standard transformer block except for the attention layer replaced with depthwise convolution.”
SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers (2021, methodology): “After that, methods such as Swin [9], CvT [58], CoaT [59], LeViT [60] and Twins [10] enhance the local continuity of features and remove ﬁxed size position embedding to improve the performance of Transformers in dense prediction tasks.”
A ConvNet for the 2020s (2022, background): “The original ViT [20] ﬁrst studied a hybrid conﬁg-uration, and a large body of follow-up works focused on reintroducing convolutional priors to ViT, either in an explicit [15,16,21,82,86,88] or implicit [45] fashion.”
PVT v2: Improved baselines with Pyramid Vision Transformer (2021, methodology): “After that, Swin Transformer [23], CoaT [36], LeViT [10], and Twins [5] further improve the classiﬁcation, detection, and segmentation performance with Transformer backbones.”
Transformer in Convolutional Neural Networks (2021, background): “CoaT [19] computes attention in the channel dimension rather than in the traditional spatial dimension.”
Focal Modulation Networks (2022, background): “Afterwards, multi-scale architectures [5, 69, 78], light-weight convolution layers [74, 22, 39], local self-attention mechanisms [46, 87, 14, 79] and learnable attention weights [84] have been proposed to boost the performance and support high-resolution input.”
MedViT: A Robust Vision Transformer for Generalized Medical Image Classification (2023, methodology): “Compared to the conventional Transformer architectures that suffice to incorporate locality bias into the lower layers of Transformer architectures through tokenization and self-attention components (CvT [12], Co-Scale ConvAttentional [13], CMT [14]).”
Continual Learning with Lifelong Vision Transformer (2022, methodology): “Besides, we also compare SOTA vision transformers (ViT [21], LeViT [25], CoaT [85], and CCT [32]) with rehearsal strategy for continual learning.”
ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias (2021, background): “[88] establishes connection across different scales at the cost of heavy computation.”
Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer (2021, background): “torization (Xu et al. 2021; Wang et al. 2020), fixed local patterns (Liu et al.”
DaViT: Dual Attention Vision Transformers (2022, background): “Similarly, CoaT [70] introduces a factorized attention mechanism that works efficiently in a multi-branch transformer backbone.”
Residual Mixture of Experts (2022, background): “Triggered by ViT, a series of follow-up works [8, 9, 14, 17, 21, 25, 30, 38, 49, 50, 52, 54, 57, 59, 60] continuously refresh the image classification records and push the potential of vision transformers in computer vision to a new height.”
MPViT: Multi-Path Vision Transformer for Dense Prediction (2021, methodology): “Transformer encoders utilize factorized multi-head self-attention (MHSA) [65].”
CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attention (2021, background): “For vision transformers, CoaT (Xu et al., 2021a) uses multi-scale features at later layers of models by mixing features from different layers; CViT (Chen et al., 2021a) keeps embeddings of different scales in different branches, and introduces cross-scale interaction through self-attention between b”
Transformer-Based Visual Segmentation: A Survey (2023, background): “There are also several works [136], [137], [138] exploring cross-scale modeling via MHSA, which exchange long-range information on different feature pyramids.”
Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations (2022, methodology): “The comparing methods include DeiT (Touvron et al., 2021a), CaiT (Touvron et al., 2021b), LVViT (Jiang et al., 2021), CoaT (Xu et al., 2021), Swin (Liu et al., 2021), Twins (Chu et al., 2021), Visformer (Chen et al., 2021), ConViT (d’Ascoli et al., 2021), TNT (Han et al., 2021a), EfficientNet (Tan &”