Refiner: Refining Self-attention for Vision Transformers

By Daquan Zhou, Yujun Shi, Bingyi Kang, Weihao Yu, Zihang Jiang, Yuan Li, Xiaojie Jin, Qibin Hou, Jiashi Feng. 2021

69 citations recorded by the source corpus.

Vision Transformers (ViTs) have shown competitive accuracy in image classification tasks compared with CNNs. Yet, they generally require much more data for model pre-training. Most of recent works thus are dedicated to designing more complex architectures or training methods to address the data-efficiency issue of ViTs. However, few of them explore improving the self-attention mechanism, a key factor distinguishing ViTs from CNNs. Different from existing works, we introduce a conceptually simple scheme, called refiner, to directly refine the self-attention maps of ViTs. Specifically, refiner explores attention expansion that projects the multi-head attention maps to a higher-dimensional space to promote their diversity. Further, refiner applies convolutions to augment local patterns of the attention maps, which we show is equivalent to a distributed local attention features are aggregated locally with learnable kernels and then globally aggregated with self-attention. Extensive experiments demonstrate that refiner works surprisingly well. Significantly, it enables ViTs to achieve 86% top-1 classification accuracy on ImageNet with only 81M parameters.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

In Model Architectures

Citations in the corpus

Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognition (2021, background): “Since then, a great number of transformer-based classification models appear, aiming at advancing the original vision transformer by either introducing locality [9], [25], [26], [27], [28], [29], or scaling the depth [8], [30], or tailoring powerful optimization strategies [7].”
MetaFormer is Actually What You Need for Vision (2021, methodology): “It is implemented by various attention mechanism in recent vision transformer models [17, 61, 66] or spatial MLP in MLP-like models [49, 50].”
Mobile-Former: Bridging MobileNet and Transformer (2021, background): “Vision transformers (ViTs): Recently, ViT [10] and its follow-ups [9, 23, 34, 35, 37, 45, 50, 51] achieve impressive performance on multiple vision tasks.”
A Survey on Vision Transformer (2020, background): “Refiner [287] explores attention expansion in higher-dimension space and applied convolution to augment local patterns of the attention maps.”
Large language models in healthcare and medical domain: A review (2023, background): “Given an input sequence X of length N , the self-attention mechanism [25, 26, 27] computes attention scores A ( i, j ) between all token pairs ( i, j ) .”
A Survey of Visual Transformers (2021, methodology): “4) The deep Transformer, such as Reﬁned-ViT [37] and CaiT [42], has great potential.”
Focal Modulation Networks (2022, methodology): “Second, we make our FocalNets deeper but thinner as in [18, 96].”
CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation (2021, background): “2021) propose DeiT that introduces a distillation strategy for transformers to help with ViT training; many other ViT variants (Yuan et al. 2021a; Wang et al. 2021b; Han et al. 2021a; Chen et al. 2021; Ranftl, Bochkovskiy, and Koltun 2021; Liu et al. 2021; Chen, Fan, and Panda 2021; Zhang et al. 202”
MetaFormer Baselines for Vision (2022, background): “As such, many attention-based token mixers [82, 21, 73, 45, 88] have been proposed in the aim to strengthen the Vision Transformers (ViTs) [17].”
Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer (2021, methodology): “Following Reﬁner (Zhou et al. 2021), we use centered kernel alignment (CKA) similarity (Kornblith et al. 2019) to measure similarity of the intermediate token features in each layer and the ﬁ-nal CLS token, assuming that the ﬁnal CLS token is strongly correlated with classiﬁcation.”