Rethinking and Improving Relative Position Encoding for Vision Transformer

By Kan Wu, Houwen Peng, Minghao Chen, Jianlong Fu, Hongyang Chao. IEEE International Conference on Computer Vision, 2021

450 citations recorded by the source corpus.

Relative position encoding (RPE) is important for transformer to capture sequence ordering of input tokens. General efficacy has been proven in natural language processing. However, in computer vision, its efficacy is not well studied and even remains controversial, e.g., whether relative position encoding can work equally well as absolute position? In order to clarify this, we first review existing relative position encoding methods and analyze their pros and cons when applied in vision transformers. We then propose new relative position encoding methods dedicated to 2D images, called image RPE (iRPE). Our methods consider directional relative distance modeling as well as the interactions between queries and relative position embeddings in self-attention mechanism. The proposed iRPE methods are simple and lightweight. They can be easily plugged into transformer blocks. Experiments demonstrate that solely due to the proposed encoding methods, DeiT [21] and DETR [1] obtain up to 1.5% (top-1 Acc) and 1.3% (mAP) stable improvements over their original versions on ImageNet and COCO respectively, without tuning any extra hyperparameters such as learning rate and weight decay. Our ablation and analysis also yield interesting findings, some of which run counter to previous understanding. Code and models are open-sourced at https://github.com/microsoft/Cream/tree/main/iRPE.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/a9c214e846188adb645021cd7b1964b8ea1fef6f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

MetaFormer is Actually What You Need for Vision (2021, background): “A lot of works have been focusing on improving the token mixing approach of transformers by shifted windows [35], relative position encoding [59], refining attention map [66], or incorporating convolution [12, 20, 58], etc.”
PETR: Position Embedding Transformation for Multi-View 3D Object Detection (2022, background): “In transformer, the features are usually added with position embedding, which provides the position information of the sequence [15,47,11,10,54], image [13,53,27] and video [1,24,52].”
TinyViT: Fast Pretraining Distillation for Small Vision Transformers (2022, background): “The attention biases [24] and a 3 × 3 depthwise convolution between attention and MLP are introduced to capture local information [69,15].”
Swin Transformer V2: Scaling Up Capacity and Resolution (2021, background): “There are also a few works particularly study how to set and learn the bias terms [38,69].”
DilateFormer: Multi-Scale Dilated Transformer for Visual Recognition (2023, methodology): “sufficient training data [22], [101] and strong data augmentation strategies [37], [71], [73], [100], [103], [109], Transformerbased methods [48], [58], [81], [86], [98], [99], [102], [105] achieve exciting performance improvements on various vision tasks, i.”
DaViT: Dual Attention Vision Transformers (2022, background): “Similarly, follow-up works [25,49,56,47,59,20,73,43,44] model global relationships on the patch-level tokens.”
Modeling Image Composition for Complex Scene Generation (2022, background): “Since autoregressive transformers have challenges in handling spatial positions [40], they may not accurately model object-object and object-patch relationships.”
Crossformer: Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting (2023, methodology): “…2015), 2) LSTnet (Lai et al., 2018), 3) MTGNN (Wu et al., 2020), and recent Transformer-based models for MTS forecasting: 4) Transformer (Vaswani et al., 2017), 5) Informer (Zhou et al., 2021), 6) Autoformer (Wu et al., 2021a), 7) Pyraformer (Liu et al., 2021a) and 8) FEDformer (Zhou et al., 2022).”
AutoFormer: Searching Transformers for Visual Recognition (2021, background): “Recent study [9] also has similar observation suggesting that relative position embedding enhances the performance of vision transformer.”
Stratified Transformer for 3D Point Cloud Segmentation (2022, methodology): “To make better use of the position information, we adopt a context-based adaptive relative position encoding scheme inspired by [51].”
A Survey on Vision Transformer (2020, background): “, positional encoding [44], [242], normalization strategy [220], shortcut connection [215] and removing attention [217], [158], [79], [218].”
Flexible Diffusion Modeling of Long Videos (2022, background): “be improved by using a novel form of relative position encoding [26, 37].”
Vision GNN: An Image is Worth Graph of Nodes (2022, background): “The main improvements include pyramid architecture [54, 33], local attention [14, 33] and position encoding [58].”