HRFormer: High-Resolution Transformer for Dense Prediction

By Yuhui Yuan, Rao Fu, Lang Huang, Weihong Lin, Chao Zhang, Xilin Chen, Jingdong Wang. arXiv.org, 2021

330 citations recorded by the source corpus.

We present a High-Resolution Transformer (HRFormer) that learns high-resolution representations for dense prediction tasks, in contrast to the original Vision Transformer that produces low-resolution representations and has high memory and computational cost. We take advantage of the multi-resolution parallel design introduced in high-resolution convolutional networks (HRNet), along with local-window self-attention that performs self-attention over small non-overlapping image windows, for improving the memory and computation efficiency. In addition, we introduce a convolution into the FFN to exchange information across the disconnected image windows. We demonstrate the effectiveness of the High-Resolution Transformer on both human pose estimation and semantic segmentation tasks, e.g., HRFormer outperforms Swin transformer by $1.3$ AP on COCO pose estimation with $50\%$ fewer parameters and $30\%$ fewer FLOPs. Code is available at: https://github.com/HRNet/HRFormer.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/e15fdbde1d56a80743b7d7eafb3409a0a5870094

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Chunhua Shen
3. Yu Qiao
4. Jing Zhang
5. Hongsheng Li
6. Jingdong Wang
7. Xinggang Wang
8. Lu Yuan

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Luc Van Gool
3. Xiaogang Wang
4. Chunhua Shen
5. Yu Qiao
6. Ming-Hsuan Yang
7. Chen Change Loy
8. Yi Yang

Citations in the corpus

• UniFormer: Unifying Convolution and Self-Attention for Visual Recognition (2022, methodology): “These works successfully boost the performance of ViT on various image tasks [8], [11], [14], [37], [45], [54], [56], [105], [109], [113], [122].”
• A Survey of Visual Transformers (2021, methodology): “Following [193], HRFormer [93] introduces the advantages of multiresolution to the Transformer along with non-overlapping local self-attention.”
• ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation (2022, methodology): “8 - HRFormer-B [48] HRFormer-B 43 78 384x288 1/4 77.”
• Multi-Scale High-Resolution Vision Transformer for Semantic Segmentation (2021, methodology): “HRFormer [38] improves HRNetV2 by replacing residual blocks with Swin Trans-former blocks.”
• DaViT: Dual Attention Vision Transformers (2022, methodology): “To further keep our architecture clean and efficient, we do not use additional operators like “Overlapping Patch” [58,55] and “ConvFFN” [72,61].”
• Depth Pro: Sharp Monocular Metric Depth in Less Than a Second (2024, unknown): “Another line of work modified the ViT architecture to produce a hierarchy of features (Fan et al., 2021; Xie et al., 2021; Yuan et al., 2021; Ranftl et al., 2021; Chen et al., 2021; Lee et al., 2022).”
• Activating More Pixels in Image Super-Resolution Transformer (2022, background): “Although vision Transformer has shown its superiority on modeling long-range dependency [13,43], there are still many works demonstrating that the convolution can help Transformer achieve better visual representation [56,58,61,60,25].”
• Vision Transformer with Deformable Attention (2022, methodology): “Without inserting convolutions in Transformer blocks [13, 14, 35], or using overlapped convolutions in patch embeddings [6, 11, 45], DATs achieve gains of +0.7, +0.7 and +0.5 over Swin Transformer [26] counterparts.”
• Rethinking Semantic Segmentation: A Prototype View (2022, unknown): “4 OCR [ECCV20] [129] HRFormer-B [130] 70.”
• TopFormer: Token Pyramid Transformer for Mobile Semantic Segmentation (2022, methodology): “To improve the efficiency, some recent works, e.g. , Swin Transformer [28], Shuffle Transformer [19], Twins [7] and HR-Former [51], compute self-attention within the lo-cal/windowed region.”
• MPViT: Multi-Path Vision Transformer for Dense Prediction (2021, methodology): “We provide a full summary of comparisons on ImageNet-1K classification in Table 8 by adding more recent Vision Transformers including ViL [71], TnT [21], ViTAE [66], HRFormer [69], and Twins [10].”