Blending Anti-Aliasing into Vision Transformer

By Shengju Qian, Hao Shao, Yi Zhu, Mu Li, Jiaya Jia. 2021

29 citations recorded by the source corpus.

The transformer architectures, based on self-attention mechanism and convolution-free design, recently found superior performance and booming applications in computer vision. However, the discontinuous patch-wise tokenization process implicitly introduces jagged artifacts into attention maps, arising the traditional problem of aliasing for vision transformers. Aliasing effect occurs when discrete patterns are used to produce high frequency or continuous information, resulting in the indistinguishable distortions. Recent researches have found that modern convolution networks still suffer from this phenomenon. In this work, we analyze the uncharted problem of aliasing in vision transformer and explore to incorporate anti-aliasing properties. Specifically, we propose a plug-and-play Aliasing-Reduction Module(ARM) to alleviate the aforementioned issue. We investigate the effectiveness and generalization of the proposed method across multiple tasks and various vision transformer families. This lightweight design consistently attains a clear boost over several famous structures. Furthermore, our module also improves data efficiency and robustness of vision transformers.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

Citations in the corpus

On Buggy Resizing Libraries and Surprising Subtleties in FID Calculation (2021, background): “While it was replaced by operations such as max-pooling, based on empirical performance [65], recent works have demonstrated that antialiasing can be compatible and improve performance in convolutional networks [79,85], transformers [57], NeRFs [4], and GANs [36].”
On Aliased Resizing and Surprising Subtleties in GAN Evaluation (2022, background): “While it was replaced by operations such as max-pooling, based on empirical performance [60], recent works have demonstrated that antialiasing can be compatible and improve performance in convolutional networks [73,79], transformers [52], NeRFs [3], and GANs [32].”
Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer (2022, background): “Later generations move on to generalize Transformer to other computer vision tasks [21, 22, 23, 24] or digging deeper to perform better [25, 26].”