CAT: Cross Attention in Vision Transformer

By Hezheng Lin, Xingyi Cheng, Xiangyu Wu, Fan Yang, Dong Shen, Zhongyuan Wang, Wei Yuan. IEEE International Conference on Multimedia and Expo, 2021

316 citations recorded by the source corpus.

Since Transformer has found widespread use in NLP, the potential of Transformer in CV has been realized and has inspired many new approaches. However, the computation required for replacing word tokens with image patches for Transformer after the tokenization of the image is vast(e.g., ViT), which bottlenecks model training and inference. In this paper, we propose a new attention mechanism in Transformer termed Cross Attention, which alternates attention inner the image patch instead of the whole image to capture local information and apply attention between image patches which are divided from single-channel feature maps to capture global information. Both operations have less computation than standard self-attention in Transformer. Based on that, we build a hierarchical network called Cross Attention Transformer(CAT) for other vision tasks. Our model achieves 82.8% on ImageNet-1K, and improves the performance of other methods on COCO and ADE20K, illustrating that our network has the potential to serve as general backbones. The code and models are avalible at https://github.com/linhezheng19/CAT.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/ab70c5e1a338cb470ec39c22a4f10e0f19e61edd

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Deng Cai
2. Xiaofei He
3. Long Chen
4. Lulian Yao
5. Yuda Song
6. Binbin Lin
7. Hui Qian
8. Xin Du

In Computer Vision and Pattern Recognition

1. Deng Cai
2. Xiaofei He
3. Dacheng Tao
4. Wei David Liu
5. Luc Van Gool
6. Yi Yang
7. Shuicheng Yan
8. Long Chen

Citations in the corpus

• SwinNet: Swin Transformer Drives Edge-Aware RGB-D and RGB-T Salient Object Detection (2022, background): “CAT [70] alternately applies attention inner patch and between patches to maintain the performance with lower computational cost and builds a cross attention hierarchical network.”
• Vision Transformers for Single Image Dehazing (2022, methodology): “Subsequently, many modified architectures [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41] have been proposed, and vision Transformer is challenging the dominance of CNNs in high-level vision tasks.”
• CrossFormer: A Versatile Vision Transformer Based on Cross-scale Attention (2021, methodology): “Thus, many substitutes (Wang et al., 2021; Liu et al., 2021b; Lin et al., 2021) are proposed to approximate the vanilla self-attention module with a lower cost.”
• CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attention (2021, methodology): “The competitors are all competitive vision transformers, including DeiT (Touvron et al., 2021), PVT (Wang et al., 2021), T2T-ViT (Yuan et al., 2021), TNT (Han et al., 2021), CViT (Chen et al., 2021a), Twins (Chu et al., 2021), Swin (Liu et al., 2021b), NesT (Zhang et al., 2021b), CvT (Wu et al., 202”