Selective Kernel Networks

By Xiang-Yang Li, Wenhai Wang, Xiaolin Hu, Jian Yang. 2019

2,712 citations recorded by the source corpus.

In standard Convolutional Neural Networks (CNNs), the receptive fields of artificial neurons in each layer are designed to share the same size. It is well-known in the neuroscience community that the receptive field size of visual cortical neurons are modulated by the stimulus, which has been rarely considered in constructing CNNs. We propose a dynamic selection mechanism in CNNs that allows each neuron to adaptively adjust its receptive field size based on multiple scales of input information. A building block called Selective Kernel (SK) unit is designed, in which multiple branches with different kernel sizes are fused using softmax attention that is guided by the information in these branches. Different attentions on these branches yield different sizes of the effective receptive fields of neurons in the fusion layer. Multiple SK units are stacked to a deep network termed Selective Kernel Networks (SKNets). On the ImageNet and CIFAR benchmarks, we empirically show that SKNet outperforms the existing state-of-the-art architectures with lower model complexity. Detailed analyses show that the neurons in SKNet can capture target objects with different scales, which verifies the capability of neurons for adaptively adjusting their receptive field sizes according to the input. The code and models are available at https://github.com/implus/SKNet.

Subfields: Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Big Self-Supervised Models are Strong Semi-Supervised Learners (2020, methodology): “The largest model we train is a 152-layer ResNet [25] with 3× wider channels and selective kernels (SK) [28], a channel-wise attention mechanism that improves the parameter efficiency of the network.”
An Empirical Study of Training Self-Supervised Vision Transformers (2021, methodology): “SimCLR v2 with SK-ResNet (Selective Kernel [28], a form of attention) has a higher envelope.”
Training data-efficient image transformers & distillation through attention (2020, background): “Motivated by the success of attention-based models in Natural Language Processing [14, 49], there has been increasing interest in architectures leveraging attention mechanisms within convnets [2, 32, 58].”
Towards Real-World Prohibited Item Detection: A Large-Scale X-ray Benchmark (2021, methodology): “Inspired by the work of [17], we propose two selective attention modules to extract channel-wise and spatial attention of different feature maps in the pyramid respectively, including the Selective Channel-wise Attention module (SCA) and the Selective Spatial Attention module (SSA).”
Dynamic Neural Networks: A Survey (2021, background): “multiple branches with independent kernel sizes [134] or feature resolutions [135] are adaptively fused.”
Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions (2021, methodology): “, ResNet, ResNeXt), there are still some specific modules and operations designed for CNNs and not considered in this work, such as SE [22], SK [35], dilated convolution [73], model pruning [19], and NAS [60].”
Self-Supervised Training Enhances Online Continual Learning (2021, methodology): “SimCLR-V2 [15] explores larger models and uses Selective Kernel Networks [39] to achieve improved performance over SimCLR.”
Going deeper with Image Transformers (2021, background): “Some approaches adapt the receptive field of convolutions dynamically [15, 42].”
ResNeSt: Split-Attention Networks (2020, methodology): “26 SKNet-101 [38] 48.”
Scaling Local Self-Attention for Parameter Efficient Visual Backbones (2021, methodology): “In addition to attention over the spatial extent that we focus on, components that perform attention over channels have also been used to augment convolutional models [20, 30].”
CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification (2021, background): “have proven beneficial for many vision tasks [5, 4, 22, 21, 25, 24, 7], but such potential benefit for vision transformers remains to be validated.”
LeViT: a Vision Transformer in ConvNet’s Clothing for Faster Inference (2021, background): “[37] use the attention mechanism between branches of the network to adapt the receptive field of neurons.”
Revisiting ResNets: Improved Training and Scaling Strategies (2021, methodology): “…(2020) further improved the ResNet-50 model with additional architectural modiﬁcations such as Squeeze-and-Excitation (Hu et al., 2018), selective kernel (Li et al., 2019), and anti-alias downsampling (Zhang, 2019), while also using label smoothing, mixup, and dropblock to achieve 81.4% accuracy.”
COVID-19 Detection Through Transfer Learning Using Multimodal Imaging Data (2020, methodology): “Another path of development combining multiple base models has resulted in a number of hybrid models including AOGNet [87], PNASNet [88], AmoebaNet [89], DPN [90], HCGNet [76], GCNet [91], ThiNet [92], and SKNet [93] etc.”
Dynamic Refinement Network for Oriented and Densely Packed Object Detection (2020, background): “To become more fine-grained, SKN [20] learned to select features with different receptive fields at each position using different kernels.”
Under review as a conference paper at ICLR 2020 completely free-form sampling in image space : Spatial Transformers (2019, methodology): “Selective Kernels are, on the other hand, comparably lightweight, but aggregating activations from kernels of different size is not as compact as directly sampling the original kernel space.”
Asymmetric Contextual Modulation for Infrared Small Target Detection (2020, methodology): “For data-driven methods, we choose FPN [22], UNet [27], selective kernel (SK) networks [19] style FPN and U-Net (SK-FPN/SK-U-Net), global attention upsampling (GAU) [18] based GAU-FPN/GAU-U-Net for comparison.”
Vision Transformers for Single Image Dehazing (2022, methodology): “The SK fusion layer is inspired by SKNet [45], it is designed to fuse multiple branches using channel attention.”
TTST: A Top-k Token Selective Transformer for Remote Sensing Image Super-Resolution (2024, methodology): “2) Context Selective Attention: To grasp the different contributions of global contexts from candidates with different large spatial fields, we introduced a channel-wise selective attention mechanism [49].”