Do We Really Need Explicit Position Encodings for Vision Transformers?

By Xiangxiang Chu, Zhi Tian, Xiaolin Wei, Huaxia Xia. 2021

107 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Towards Robust Vision Transformer (2021, methodology): “To get over this shortcoming, CPVT [3] and LeViT [12] remove the CLS token and replace it by average pooling along with the last layer sequential output of the Transformer.”
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows (2021, methodology): “Concurrent to our work are some that modify the ViT architecture [66, 15, 25] for better image classification.”
CvT: Introducing Convolutions to Vision Transformers (2021, methodology): “For example, the Conditional Position encodings Visual Transformer (CPVT) [6] replaces the prede-ﬁned positional embedding used in ViT with conditional position encodings (CPE), enabling Transformers to process input images of arbitrary size without interpolation.”
Multiscale Vision Transformers (2021, background): “A separate line of works attempts at modeling visual data with learnt discretized token sequences [110, 89, 17, 115, 21].”
Transformer in Transformer (2021, background): “Nowadays, transformer architectures have been used in a growing number of computer vision tasks [10] such as image recognition [6, 40, 29], object detection [46], and segmentation [43, 38].”
LeViT: a Vision Transformer in ConvNet’s Clothing for Faster Inference (2021, background): “Ablations of the positional embedding result in a sharp drop of the classification accuracy [55].”
Residual Mixture of Experts (2022, background): “Triggered by ViT, a series of follow-up works [8, 9, 14, 17, 21, 25, 30, 38, 49, 50, 52, 54, 57, 59, 60] continuously refresh the image classification records and push the potential of vision transformers in computer vision to a new height.”
A Simple Single-Scale Vision Transformer for Object Localization and Instance Segmentation (2021, background): “More recent trends [10,18,28,41–43] start following design conventions in CNNs discussed above (Section 2.1) and customize ViT architectures to be CNN-like: tokens are progressively merged to downsample the feature resolutions with reduced computation cost, along with increased embedding sizes.”
P2T: Pyramid Pooling Transformer for Scene Understanding (2021, methodology): “P2T is also compatible with other transformer techniques such as knowledge distillation [43], patch embedding [45], positional encoding [46], and feed-forward network [23], [47].”
Co-Scale Conv-Attentional Image Transformers (2021, methodology): “The work of CPVT [4] explores the use of convolution as conditional position encodings by inserting it after the feed-forward network under a single scale ( H 16 × W16 ).”
CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows (2021, methodology): “Built upon the success of ViT, many efforts have been devoted to designing better Transformer based architectures for various vision tasks, including low-level image processing [5, 58], image classification [54, 67, 65, 14, 21, 59, 12, 61, 12, 66, 31, 55, 19, 24], object detection [3, 74] and semant”
UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning (2022, methodology): “To overcome the above problems, we extend the conditional position encoding (CPE) (Chu et al., 2021) to design our DPE: where DWConv means simple 3D depthwise convolution with zero paddings.”
DilateFormer: Multi-Scale Dilated Transformer for Visual Recognition (2023, methodology): “works [14], [22], [58], [73], [74] introduce Transformers with long-range dependency modeling ability into computer vision, achieving exciting results in various vision tasks.”