Improve Vision Transformers Training by Suppressing Over-smoothing

By Chengyue Gong, Dilin Wang, Meng Li, Vikas Chandra, Qiang Liu. 2021

30 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Jiashi Feng
2. Dacheng Tao
3. Luc Van Gool
4. Shuicheng Yan
5. Ming-Hsuan Yang
6. Yi Yang
7. Xiaogang Wang
8. Trevor Darrell

In Model Architectures

1. Jiashi Feng
2. Qibin Hou
3. Guiguang Ding
4. Weihao Yu
5. Anima Anandkumar
6. Jian‐Yuan Sun
7. Zihang Jiang
8. Jie Zhou

In Optimization & Training

1. Jiashi Feng
2. Anima Anandkumar
3. Qibin Hou
4. Tom Goldstein
5. Bryan Catanzaro
6. Jian‐Yuan Sun
7. Jie Zhou
8. Jiwen Lu

Citations in the corpus

• Global Filter Networks for Image Classification (2021, background): “These works either seek for better training strategies [43, 11] or design better architectures [29, 48, 52] or both [44, 11].”
• Scaling Up Your Kernels to 31×31: Revisiting Large Kernel Design in CNNs (2022, methodology): “Even equipped with DiverseP-atch [38], a technique customized for vision transformers, the single-scale mIoU of the 22K-pretrained Swin-L is still lower than our 1K-pretrained RepLKNet-31B, though the former has 2 × parameters.”
• Refiner: Refining Self-attention for Vision Transformers (2021, background): “Though benefiting in learning flexible features from large-scale samples, such global-range attention may lead to overly-smoothed attention maps and token features as pointed out by [66, 21].”
• Long-Short Transformer: Efficient Transformers for Language and Vision (2021, methodology): “As in [58], we provide the cosine similarity of patch embeddings of our CvT∗-LS-13 and re-implemented CvT-13 (81.”