ResNeSt: Split-Attention Networks

By Hang Zhang, Chongruo Wu, Zhongyue Zhang, Yi Zhu, Zhi-Li Zhang, Haibin Lin, Yue Sun, Tong He, Jonas W. Mueller, Raghavan Manmatha and 2 more. 2020

1,747 citations recorded by the source corpus.

The ability to learn richer network representations generally boosts the performance of deep learning models. To improve representation-learning in convolutional neural networks, we present a multi-branch architecture, which applies channel-wise attention across different network branches to leverage the complementary strengths of both feature-map attention and multi-path representation. Our proposed Split-Attention module provides a simple and modular computation block that can serve as a drop-in replacement for the popular residual block, while producing more diverse representations via cross-feature interactions. Adding a Split-Attention module into the architecture design space of RegNet-Y and FBNetV2 directly improves the performance of the resulting network. Replacing residual blocks with our Split-Attention module, we further design a new variant of the ResNet model, named ResNeSt, which outperforms EfficientNet in terms of the accuracy/latency trade-off.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Luc Van Gool
2. Quoc V. Le
3. Jiashi Feng
4. Jianfeng Gao
5. Dacheng Tao
6. Alan L. Yuille
7. Trevor Darrell
8. Bernt Schiele

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Ming-Hsuan Yang
4. Xiaogang Wang
5. Trevor Darrell
6. Alan L. Yuille
7. Yi Yang
8. Jitendra Malik

In Model Architectures

1. Quoc V. Le
2. Jiashi Feng
3. Shuicheng Yan
4. Jianfeng Gao
5. Lu Yuan
6. Alan L. Yuille
7. Mingxing Tan
8. Liang-Chieh Chen

Citations in the corpus

• Training data-efficient image transformers & distillation through attention (2020, background): “Motivated by the success of attention-based models in Natural Language Processing [14, 49], there has been increasing interest in architectures leveraging attention mechanisms within convnets [2, 32, 58].”
• Language-driven Semantic Segmentation (2022, neutral): “We compare the results to highly competitive standard semantic segmentation models, including OCNet (Yuan et al., 2020), ACNet (Fu et al., 2019), DeeplabV3 (Chen et al., 2017; Zhang et al., 2020a), and DPT (Ranftl et al., 2021).”
• Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation (2020, methodology): “In-stance segmentation models based on state-of-the-art convolutional networks [11, 57, 67] are often data-hungry.”
• Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions (2021, neutral): “Moreover, with years of rapid developments, there have been many well-engineered CNN backbones such as Res2Net [17], EfficientNet [61], and ResNeSt [79].”
• Going deeper with Image Transformers (2021, methodology): “Various other attention mechanism have been used successfully to give a global view in conjunction with (local) convolutions [4, 52, 12, 77, 79], most mimic squeeze-and-excitate [32] for leveraging global features.”
• Segmenter: Transformer for Semantic Segmentation (2021, methodology): “Transformer versus FCN. Table 4 and Table 6 compare our approach to FCN models and DeepLabv3+ [10] with ResNeSt backbone [63], one of the best fully-convolutional approaches.”
• SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers (2021, methodology): “Indeed, starting from early methods using VGGs [1, 2], to the latest methods with significantly deeper and more powerful backbones [3], the evolution of backbones has dramatically pushed the performance boundary of semantic segmentation.”
• Bottleneck Transformers for Visual Recognition (2021, methodology): “This result surpasses the previous best published single model single scale instance segmentation result from ResNeSt [67] evaluated on the COCO minival (44.2% AP mk ).”
• Vision Transformers for Dense Prediction (2021, neutral): “[51].”
• EfficientNetV2: Smaller Models and Faster Training (2021, background): “We would like to highlight a few interesting observations: • Scaling up data size is more effective than simply scaling up model size in high-accuracy regime : when the top-1 accuracy is beyond 85%, it is very difficult to further improve it by simply increasing model size (Radosavovic et al., 2020) ”
• DeepViT: Towards Deeper Vision Transformer (2021, methodology): “The recent progress of deep CNN models is largely driven by training very deep models with a large number of layers which is enabled by novel model architecture designs [9, 41, 31, 20, 47].”
• LeViT: a Vision Transformer in ConvNet’s Clothing for Faster Inference (2021, background): “The mechanism is used channel-wise to capture cross-feature information that complements convolutional layers [39, 40, 41], select paths in different branch of a network [42], or combine both [43].”
• Revisiting ResNets: Improved Training and Scaling Strategies (2021, methodology): “Other works such as ResNeSt (Zhang et al., 2020) and TResNet (Ridnik et al., 2020)) scale the image resolution up to 448.”