HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions

By Yongming Rao, Wenliang Zhao, Yansong Tang, Jie Zhou, Ser-Nam Lim, Jiwen Lu. Neural Information Processing Systems, 2022

401 citations recorded by the source corpus.

Recent progress in vision Transformers exhibits great success in various tasks driven by the new spatial modeling mechanism based on dot-product self-attention. In this paper, we show that the key ingredients behind the vision Transformers, namely input-adaptive, long-range and high-order spatial interactions, can also be efficiently implemented with a convolution-based framework. We present the Recursive Gated Convolution ($\textit{g}^\textit{n}$Conv) that performs high-order spatial interactions with gated convolutions and recursive designs. The new operation is highly flexible and customizable, which is compatible with various variants of convolution and extends the two-order interactions in self-attention to arbitrary orders without introducing significant extra computation. $\textit{g}^\textit{n}$Conv can serve as a plug-and-play module to improve various vision Transformers and convolution-based models. Based on the operation, we construct a new family of generic vision backbones named HorNet. Extensive experiments on ImageNet classification, COCO object detection and ADE20K semantic segmentation show HorNet outperform Swin Transformers and ConvNeXt by a significant margin with similar overall architecture and training configurations. HorNet also shows favorable scalability to more training data and larger model sizes. Apart from the effectiveness in visual encoders, we also show $\textit{g}^\textit{n}$Conv can be applied to task-specific decoders and consistently improve dense prediction performance with less computation. Our results demonstrate that $\textit{g}^\textit{n}$Conv can be a new basic module for visual modeling that effectively combines the merits of both vision Transformers and CNNs. Code is available at https://github.com/raoyongming/HorNet

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/3e448df5aa191f7a3945d0fd609c8bc5966a2333

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

MetaFormer Baselines for Vision (2022, methodology): “The power of MetaFormer can also be verified by the recent models adopting MetaFormer as the general architecture but with different attention-based [40, 58, 81], MLP-based [67, 68, 27, 6, 41], convolution-based [15, 19, 80, 56, 78], hybrid [62, 12, 37, 71] or other types of [65, 20] token mixers.”
Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks (2022, background): “Model Crop Size ADE20K mIoU +MS HorNet [RZT22] 640(2) 57.”
Fast Vision Transformers with HiLo Attention (2022, background): “In Figure 6, we show that HiLo is consistently faster than many attention mechanisms [51, 32, 1, 50, 11, 17, 55, 60, 41, 44, 20, 18] on both CPUs and GPUs.”
Rewrite the Stars (2024, neutral): “GLU [48], etc. ), Computer Vision ( i.e. , FocalNet [60], Hor-Net [45], VAN [18], etc. ), and more.”
InceptionNeXt: When Inception Meets ConvNeXt (2023, neutral): “ConvNeXt is not alone: More and more works have shown similar observations [14, 18, 24, 38, 51, 68, 76, 78], like RepLKNet [14] and SLaK [38].”
VMamba: Visual State Space Model (2024, neutral): “VMamba contributes as an alternative foundation model to the community, alongside CNNs and ViTs.”
MambaOut: Do We Really Need Mamba for Vision?* (2024, neutral): “…under the spirit of RNN. Neverthe-less, their experiments show that the SSM based models for vision, in reality, lead to underwhelming performance compared with state-of-the-art convolutional [23, 31, 39, 54, 58, 70, 86, 98, 100, 101] and attention-based models [19, 24, 51, 76, 80, 81, 100, 103].”