CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attention

By Wenxiao Wang, Lulian Yao, Long Chen, Binbin Lin, Deng Cai, Xiaofei He, Wei David Liu. International Conference on Learning Representations, 2021

380 citations recorded by the source corpus.

Transformers have made great progress in dealing with computer vision tasks. However, existing vision transformers do not yet possess the ability of building the interactions among features of different scales, which is perceptually important to visual inputs. The reasons are two-fold: (1) Input embeddings of each layer are equal-scale, so no cross-scale feature can be extracted; (2) to lower the computational cost, some vision transformers merge adjacent embeddings inside the self-attention module, thus sacrificing small-scale (fine-grained) features of the embeddings and also disabling the cross-scale interactions. To this end, we propose Cross-scale Embedding Layer (CEL) and Long Short Distance Attention (LSDA). On the one hand, CEL blends each embedding with multiple patches of different scales, providing the self-attention module itself with cross-scale features. On the other hand, LSDA splits the self-attention module into a short-distance one and a long-distance counterpart, which not only reduces the computational burden but also keeps both small-scale and large-scale features in the embeddings. Through the above two designs, we achieve cross-scale attention. Besides, we put forward a dynamic position bias for vision transformers to make the popular relative position bias apply to variable-sized images. Hinging on the cross-scale attention module, we construct a versatile vision architecture, dubbed CrossFormer, which accommodates variable-sized inputs. Extensive experiments show that CrossFormer outperforms the other vision transformers on image classification, object detection, instance segmentation, and semantic segmentation tasks. The code has been released: https://github.com/cheerss/CrossFormer.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/a5c41f188b0eb0acb444cb4899bf6af378ee9ede

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation (2022, neutral): “Inspired by the success of ViT [14], many different vision transformer backbones [32, 45, 42, 55, 41, 52, 40, 53] have been proposed, which are typically trained on the ImageNet-1K [13] dataset in a fully supervised setting.”
Vision Transformers for Single Image Dehazing (2022, methodology): “Subsequently, many modified architectures [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41] have been proposed, and vision Transformer is challenging the dominance of CNNs in high-level vision tasks.”
CMTFNet: CNN and Multiscale Transformer Fusion Network for Remote-Sensing Image Semantic Segmentation (2023, methodology): “With the rapid development of transformers in the field of natural language processing (NLP), some researchers have tried to solve this problem by using transformers in semantic segmentation tasks [22], [23], [24], [25].”
Masked Autoencoders for Point Cloud Self-supervised Learning (2022, background): “Since ViT [12], Trans-formers architectures have been popular in computer vision [55,25,56,43,42,59,16,6].”
Multi-Agent Reinforcement Learning is a Sequence Modeling Problem (2022, positive): “Although the idea of solving CV tasks by SM is straightforward, it serves as the foundation to some of the best-performing CV algorithms [38, 41, 39].”
EfficientFormer: Vision Transformers at MobileNet Speed (2022, methodology): “To accelerate ViT, many approaches have been introduced with different methodologies, such as proposing new architectures or modules [53, 54, 55, 56, 57, 58], re-thinking self-attention and sparse-attention mechanisms [59, 60, 61, 62, 63, 64, 65], and utilizing search algorithms that are widely expl”
Rethinking Vision Transformers for MobileNet Size and Speed (2022, methodology): “For reducing the computation complexity of the vision transformer, many works propose new modules and refine architecture design [10,20,26,37,41,61], while others eliminate redundancies in attention mechanism [9, 14, 29, 44, 60, 71, 74].”
Dual Aggregation Transformer for Image Super-Resolution (2023, background): “where D denotes the relative position encoding [40].”
BiFormer: Vision Transformer with Bi-Level Routing Attention (2023, methodology): “The ﬁrst problem comes as how [27, 44], axial stripe [13], dilated window [39, 44].”