XCiT: Cross-Covariance Image Transformers

By Alaaeldin El-Nouby, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek and 1 more. 2021

669 citations recorded by the source corpus.

Following their success in natural language processing, transformers have recently shown much promise for computer vision. The self-attention operation underlying transformers yields global interactions between all tokens ,i.e. words or image patches, and enables flexible modelling of image data beyond the local interactions of convolutions. This flexibility, however, comes with a quadratic complexity in time and memory, hindering application to long sequences and high-resolution images. We propose a"transposed"version of self-attention that operates across feature channels rather than tokens, where the interactions are based on the cross-covariance matrix between keys and queries. The resulting cross-covariance attention (XCA) has linear complexity in the number of tokens, and allows efficient processing of high-resolution images. Our cross-covariance image transformer (XCiT) is built upon XCA. It combines the accuracy of conventional transformers with the scalability of convolutional architectures. We validate the effectiveness and generality of XCiT by reporting excellent results on multiple vision benchmarks, including image classification and self-supervised feature learning on ImageNet-1k, object detection and instance segmentation on COCO, and semantic segmentation on ADE20k.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Benchmarking Detection Transfer Learning with Vision Transformers (2021, methodology): “…1 / 16 th), in contrast to the multi-scale feature maps produced by typical CNNs. 1 To address this discrepancy, we employ a simple technique from [11] (used for the single-scale XCiT back-bone) to either upsample or downsample intermediate ViT feature maps by placing four resolution-modifying…”
Exploring Plain Vision Transformer Backbones for Object Detection (2022, background): “In the first variant, the backbone is artificially divided into multiple stages to mimic the stages of a hierarchical backbone, with lateral and top-down connections applied (Figure 2 (a)) [15].”
Transformers in Vision: A Survey (2021, neutral): “Cross-Covariance Image Transformers (XCiT) [59] incorporate attention across feature-channels instead”
GroupViT: Semantic Segmentation Emerges from Text Supervision (2022, methodology): “Thus, instead of organizing visual tokens into grids, as recent ViT-based applications [16, 23, 44, 81] do, we propose to perform hierarchical grouping of visual tokens into irregular-shaped segments.”
Attention mechanism in neural networks: where it comes and where it goes (2022, methodology): “Image Transformer based on the crosscovariance matrix between keys and queries is applied [187], and a new vision Transformer is proposed [188].”
Contextual Transformer Networks for Visual Recognition (2021, background): “Most recently, [35] designs a novel cross-covariance attention (XCA) which is a “transposed” version of conventional self-attention that operates across feature channels rather than tokens (words or image patches), thereby achieving linear complexity in the number of tokens.”
Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction (2021, background): “Recently, Transformer has achieved great success in many high-level vision tasks, such as image classification [2, 13, 15, 28, 40, 55], object detection [1, 11, 28, 61,”
Neighborhood Attention Transformer (2022, methodology): “Some works focused on data efficiency, with minor changes in the architecture [26, 33, 11, 27, 10], while others focused on efficiency and transferrability to downstream tasks [18, 34, 28, 10].”
CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers (2022, background): “Recent works have been focused on improving the architectures of attention models, including sparse attention [29, 30, 31, 32, 33, 34, 35], enlarged receptive ﬁelds [36, 37], pyramidal designs [38, 39, 40], efﬁcient alternatives [41, 42, 43], etc.”
DeiT III: Revenge of the ViT (2022, neutral): “Table 7: ADE20k semantic segmentation performance using UperNet [58] (in comparable settings [10,15,31]).”
Lite-Mono: A Lightweight CNN and Transformer Architecture for Self-Supervised Monocular Depth Estimation (2022, methodology): “Stage 3 CDC blocks [3× 3, 128]× 6 [3× 3, 128]× 6 [3× 3, 128]× 9 [3× 3, 224]× 9 LGFI block dilation = 1, 2, 3, 2, 4, 6 dilation = 1, 2, 3, 2, 4, 6 dilation = [1, 2, 3]× 2, 2, 4, 6 dilation = [1, 2, 3]× 2, 2, 4, 6”
ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond (2022, background): “Similar phenomena can also be observed in models with fewer inductive biases in their structures (Tolstikhin et al., 2021; Ali et al., 2021; He et al., 2021).”
ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias (2021, background): “Following works along this direction are to simplify the model structures with fewer intrinsic IBs and directly learn them from large scale data [50, 74, 75, 22, 18, 20, 27] which have achieved promising results and been studied actively.”
Visual attention network (2022, background): “Besides, self-attention is a special attention that only considers the adaptability in spatial dimension but ignores the adaptability in channel dimension, which is also important for visual tasks [33,85,80,1].”
MambaVision: A Hybrid Mamba-Transformer Vision Backbone (2024, neutral): “ViTs [5] marked a signiﬁcant shift in computer vision by introducing self-attention mechanisms for enlarged receptive ﬁelds.”
Are Large-scale Datasets Necessary for Self-Supervised Pre-training? (2021, positive): “In order to obtain features compatible with the Feature Pyramid Network (FPN) design [69], we use max pooling and transposed convolution operations similar to El-Nouby et al. [70].”
EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applications (2022, methodology): “Top1 ↑ Latency ↓ 1: Conv=[3, 3, 9, 3], SDTA=[0, 0, 0, 0] 78. els.”