Squeeze-and-Excitation Networks

By Jie Hu, Li Shen, Samuel Albanie, Gang Sun, Enhua Wu. 2017

1,495 citations recorded by the source corpus.

The central building block of convolutional neural networks (CNNs) is the convolution operator, which enables networks to construct informative features by fusing both spatial and channel-wise information within local receptive fields at each layer. A broad range of prior research has investigated the spatial component of this relationship, seeking to strengthen the representational power of a CNN by enhancing the quality of spatial encodings throughout its feature hierarchy. In this work, we focus instead on the channel relationship and propose a novel architectural unit, which we term the “Squeeze-and-Excitation” (SE) block, that adaptively recalibrates channel-wise feature responses by explicitly modelling interdependencies between channels. We show that these blocks can be stacked together to form SENet architectures that generalise extremely effectively across different datasets. We further demonstrate that SE blocks bring significant improvements in performance for existing state-of-the-art CNNs at slight additional computational cost. Squeeze-and-Excitation Networks formed the foundation of our ILSVRC 2017 classification submission which won first place and reduced the top-5 error to 2.251 percent, surpassing the winning entry of 2016 by a relative improvement of <inline-formula><tex-math notation="LaTeX">${\sim }$</tex-math><alternatives><mml:math><mml:mo>∼</mml:mo></mml:math><inline-graphic xlink:href="shen-ieq1-2913372.gif"/></alternatives></inline-formula>25 percent. Models and code are available at <uri>https://github.com/hujie-frank/SENet</uri>.

Subfields: Artificial Intelligence, Atomic and Molecular Physics, and Optics, Computer Vision and Pattern Recognition, Molecular Biology, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Atomic and Molecular Physics, and Optics

In Computer Vision and Pattern Recognition

In Molecular Biology

In Model Architectures

Citations in the corpus

Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need? (2020, methodology): “ality of embeddings, we compare three alternatives in Table5and Table6: a ConvNet with four four convolutional layers (64, 64, 64, 64); a ResNet12 as in Table1; a ResNet12 with sequeeze-andexcitation [19] modules. For each model, we have four settings: training on meta-training set; training and dis”
Progressive Neural Architecture Search (2017, background): “best-performing CIFAR-10 model. Table 5 shows that under the Large setting, PNASNet-5 achieves higher performance (82.9% top-1; 96.2% top-5) than previous state-of-the-art approaches, including SENet [35], NASNet-A, and AmoebaNets under the same model capacity. 6 Discussion and Future Work The main ”
BAM: Bottleneck Attention Module (2018, methodology): “A few attempts [19, 42] have been made to consider attention as an effective solution for general classiﬁcation task.”
Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations (2017, background): “In Szegedy et al. (2017), Zhang et al. (2017), Xie et al. (2017), Li et al. (2017) and Hu et al. (2017), the authors further improved the accuracy of the networks by carefully designing residual blocks via increasing the width of each block, changing the topology of the network and following…”
fastai: A Layered API for Deep Learning (2020, methodology): “tions from ResNet [40], and therefore this class is called XResNet. By providing parameters to this class, the user can customise it to create architectures that include squeeze and excitation blocks [42], grouped convolutions such as in ResNext [43], depth-wise convolutions such as in the Xception ”
Deep learning in bioinformatics: introduction, application, and perspective in big data era (2019, methodology): “l the mapping between the input and the output, here in (C), those convolutional layers are used to model the residual between the input and the residual block output. (D) The building block of SENet [58]. This architecture further improves the residual block, which introduces the idea of Attention ”
An investigation of model-free planning (2019, methodology): “This is operation is related to the pool-and-inject method introduced by Racanière et al. (2017) and to Squeeze-and-Excitation blocks (Hu et al., 2017).”
ACNET: Attention Based Network to Exploit Complementary Features for RGBD Semantic Segmentation (2019, background): “In order to gather features selectively from RGB branch and depth branch, we have designed a set of attention modules [11] to make the network focus on more informative regions.”
Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification (2020, background): “te-ofthe-art ﬁlter pruning methods. 1. Introduction The booming development in deep learning models and applications has enabled beyond human performance in tasks like large-scale image classiﬁcation [18,9,12,13], object detection [27,22,8], and semantic segmentation [24, 3]. Such high performance, ”
Geometric Back-Projection Network for Point Cloud Classification (2019, methodology): “ds, another widely applied mechanism: Attention, can assist the network to put more emphasis on useful information [36]. Attention modules are used in many 2D visual problems (e.g. image segmentation [42,8,7], image denoising [1]), etc. For 3D point cloud analysis, we design a Channel-wise Afﬁnity A”
BACH: Grand challenge on breast cancer histology images (2018, methodology): “ttention mechanism for dynamic channel selection following the design trends of DenseNet, Squeeze-Excitation Network (SENet) and ResNext (Huang et al., 2017; J´egou et al., 2016; Yu and Koltun, 2015; Hu et al., 2017; Xie et al., 2017). For classifying the microscopy images, the model has a head comp”
SCRDet: Towards More Robust Detection for Small, Cluttered and Rotated Objects (2018, methodology): “Many attention structures [18, 17, 37, 38] have been proposed to solve problems of occlusion, noise, and blurring.”
Object Detection in 20 Years: A Survey (2019, methodology): “GoogLeNet : GoogLeNet, a.k.a Inception [198, 231–233], is a big family of CNN models proposed by Google Inc. SENet : Squeeze and Excitation Networks (SENet) was proposed by J. Hu and L. Shen et al. in 2018 [236].”
Multivariate LSTM-FCNs for Time Series Classification (2018, background): “Hu et al. [30] propose a Squeeze-and-Excitation block that acts as a computational unit for any transformation The convolution operation is depicted by *, and the 2D spatial kernel is depicted by v sc .”
Zoom Out-and-In Network with Map Attention Decision for Region Proposal and Object Detection (2017, methodology): “ER vector denotes the Excitation Recal-ibration idea implemented in SENet (Hu et al., 2017).”