Squeeze-and-Excitation Networks
By Jie Hu, Li Shen, Samuel Albanie, Gang Sun, Enhua Wu. 2017
35,229 citations recorded by the source corpus.
Convolutional neural networks are built upon the convolution operation, which extracts informative features by fusing spatial and channel-wise information together within local receptive fields. In order to boost the representational power of a network, several recent approaches have shown the benefit of enhancing spatial encoding. In this work, we focus on the channel relationship and propose a novel architectural unit, which we term the "Squeeze-and-Excitation" (SE) block, that adaptively recalibrates channel-wise feature responses by explicitly modelling interdependencies between channels. We demonstrate that by stacking these blocks together, we can construct SENet architectures that generalise extremely well across challenging datasets. Crucially, we find that SE blocks produce significant performance improvements for existing state-of-the-art deep architectures at minimal additional computational cost. SENets formed the foundation of our ILSVRC 2017 classification submission which won first place and significantly reduced the top-5 error to 2.251%, achieving a ~25% relative improvement over the winning entry of 2016. Code and models are available at https://github.com/hujie-frank/SENet.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Jianfeng Gao
- Yoshua Bengio
- Mohit Bansal
- Dacheng Tao
- Trevor Darrell
- Sergey Levine
- Percy Liang
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Xiaogang Wang
- Trevor Darrell
- Jitendra Malik
- Yi Yang
- Alan L. Yuille
In Model Architectures
- Quoc V. Le
- Alan L. Yuille
- Jianfeng Gao
- Jiashi Feng
- Yoshua Bengio
- Dacheng Tao
- Shuicheng Yan
- Luc Van Gool
Citations in the corpus
- EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks (2019, methodology): “We omit ensemble and multi-crop models (Hu et al., 2018), or models pretrained on 3.5B Instagram images (Mahajan et al., 2018).”
- Visual Transformers: Token-based Image Representation and Processing for Computer Vision (2020, methodology): “We compare our results with other previous methods [7, 9, 16, 38, 15, 18, 17] that adopt attention in different ways on ResNets.”
- Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, neutral): “05 SE-ResNet-101 [39] 15.”
- On the Relationship between Self-Attention and Convolutional Layers (2019, methodology): “Self-attention was first added to CNN by either using channel-based attention (Hu et al., 2018) or non-local relationships across the image (Wang et al., 2018).”
- Local Relation Networks for Image Recognition (2019, background): “In [13], the channel-wise attention are explored and [30] further integrated the channel-wise attention and spatial-wise attention.”
- Attention Augmented Convolutional Networks (2019, methodology): “In all experiments, Attention Augmentation sig-nificantly increases performance over the non-augmented baseline and notably outperforms Squeeze-and-Excitation (SE) [17] while being more parameter efficient (Figure 1).”
- CCNet: Criss-Cross Attention for Semantic Segmentation (2018, background): “Squeeze-and-Excitation Networks [18] enhanced the representational power of the network by modeling channel-wise relationships in an attention mechanism.”
- Natural Adversarial Examples (2019, methodology): “We consider the self-attention technique called Squeeze-andExcitation (SE) [24], which won the final ImageNet competition.”
- Do ImageNet Classifiers Generalize to ImageNet? (2019, neutral): “se_resnext101_32x4d [26] https://github.”
- Bag of Tricks for Image Classification with Convolutional Neural Networks (2018, methodology): “of other models, such as SENet [12], PSPNet [32], DeepLabV3 [1], and ShuffleNetV2 [21].”
- Self-supervised Pretraining of Visual Features in the Wild (2021, methodology): “In this work, we focus on the RegNetY architectures, that add a Squeeze-and-excitation op [26] to the standard RegNets to further improve their performance.”