Conditional Positional Encodings for Vision Transformers

By Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, Chunhua Shen. 2021

838 citations recorded by the source corpus.

We propose a conditional positional encoding (CPE) scheme for vision Transformers. Unlike previous fixed or learnable positional encodings, which are pre-defined and independent of input tokens, CPE is dynamically generated and conditioned on the local neighborhood of the input tokens. As a result, CPE can easily generalize to the input sequences that are longer than what the model has ever seen during training. Besides, CPE can keep the desired translation-invariance in the image classification task, resulting in improved performance. We implement CPE with a simple Position Encoding Generator (PEG) to get seamlessly incorporated into the current Transformer framework. Built on PEG, we present Conditional Position encoding Vision Transformer (CPVT). We demonstrate that CPVT has visually similar attention maps compared to those with learned positional encodings and delivers outperforming results. Our code is available at https://github.com/Meituan-AutoML/CPVT .

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Transformers in Vision: A Survey (2021, background): “Twins [42] builds upon PVT [210] (an attention only pyramid design), by replacing the absolute position embedding in PVT with relative conditional position embedding [43], and incorporating the separable depth-wise convolutions instead of the standard spatial attention, to capture local and global c”
Focal Self-attention for Local-Global Interactions in Vision Transformers (2021, background): “Apart from the downstream tasks, another line of work focus on improving the original vision transformers from different perspectives, such as data-efficient training [57], improved patch embedding/encoding [18, 75, 32], integrating convolutional projections into transformers [67, 74], multi-scale a”
PVT v2: Improved baselines with Pyramid Vision Transformer (2021, methodology): “CPVT [6] replaces the ﬁxed size position embedding in ViT with conditional position encodings, making it easier to process images of arbitrary resolution.”
DilateFormer: Multi-Scale Dilated Transformer for Visual Recognition (2023, methodology): “To make the position encoding adaptive to inputs of different resolutions, we use Conditional Position Embedding (CPE) proposed in CPVT [24] whenever inputs are fed into MSDA or MHSA blocks.”
Multi-class Token Transformer for Weakly Supervised Semantic Segmentation (2022, methodology): “Although the class token has been removed in a number of recent transformer methods [7, 8, 29], this work will underline its importance for weakly supervised semantic segmentation.”
Blending Anti-Aliasing into Vision Transformer (2021, background): “For example, CPVT [25] and CoaT [26] improves the original positional encoding.”
DaViT: Dual Attention Vision Transformers (2022, methodology): “We also simplifies its relative position encoding [10] with a depth-wise convolution before layernorm to get a cleaner structure for arbitrary input sizes.”
FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterization (2023, methodology): “Positional Encodings We use conditional positional encodings [7, 8] that is dynamically generated and conditioned on the local neighborhood of the input tokens.”
Twins: Revisiting the Design of Spatial Attention in Vision Transformers (2021, methodology): “We use the simplest form of PEG, i.e. , a 2D depth-wise convolution without batch normalization.”
Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer (2021, methodology): “To enhance local context in Vision Transformers, the Conditional Position encodings Visual Transformer (CPVT) [9] replaces the predeﬁned positional embedding used in ViT with conditional position encodings (CPE), enabling Transformers to process input images of arbitrary size without interpolation.”
A Survey on Vision Transformer (2020, background): “, positional encoding [44], [242], normalization strategy [220], shortcut connection [215] and removing attention [217], [158], [79], [218].”
Stripformer: Strip Transformer for Fast Image Deblurring (2022, methodology): “Specifically, we use LayerNorm, feed-forward MultiLayer Perceptron (MLP) with a residual connection, and the Conditional Positional Encodings [29] (CPE), a 3 × 3 depth-wise convolution layer with a residual connection, to generate the final output Ointra ∈ RH×W×C as”
RegionViT: Regional-to-Local Attention for Vision Transformers (2021, methodology): “…49.2 − − − 119.7 474 45.9 − − 41.6 − − RegionViT-B+ w/ PEG† 84.5 506.4 46.1 68.0 49.5 30.5 49.9 60.1 93.2 464.4 46.3 69.1 51.2 42.4 66.2 45.6 The reported results of Swin (Liu et al., 2021) are from Twins (Chu et al., 2021a) as the original paper does not include resutls with ImageNet1K weights.”
Medical Image Segmentation via Cascaded Attention Decoding (2023, background): “However, the selfattention used in transformers limits their ability to learn local (contextual) relations among pixels [7, 16].”
Co-Scale Conv-Attentional Image Transformers (2021, methodology): “The work of CPVT [6] explores the use of convolution as conditional position encodings by inserting it after the feed-forward network under a single scale ( H 16 × W16 ).”
P2T: Pyramid Pooling Transformer for Scene Understanding (2021, methodology): “Besides, our design is also compatible with other transformer techniques such as patch embedding [51], positional encoding [50], and feed-forward network [24], [52], [53].”
Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition (2021, methodology): “Some following works such as T2T-ViT [55], TNT [17], CaiT [39], DeepViT [62], CPVT [6], LocalViT [28] and CrossViT [5] focus on improving the architecture design of ViT.”
CMT: Convolutional Neural Networks Meet Vision Transformers (2021, background): “For instance segmentation with Mask R-CNN as basic framework, CMT-S surpasses Twins-PCPVT-S [5] with 1.7% AP and Twins-SVT-S [5] with 1.9% AP.”