Dynamic Channel Pruning: Feature Boosting and Suppression

By Xitong Gao, Yiren Zhao, Lukasz Dudziak, Robert D. Mullins, Chengzhong Xu. 2018

348 citations recorded by the source corpus.

Making deep convolutional neural networks more accurate typically comes at the cost of increased computational and memory resources. In this paper, we reduce this cost by exploiting the fact that the importance of features computed by convolutional layers is highly input-dependent, and propose feature boosting and suppression (FBS), a new method to predictively amplify salient convolutional channels and skip unimportant ones at run-time. FBS introduces small auxiliary connections to existing convolutional layers. In contrast to channel pruning methods which permanently remove channels, it preserves the full network structures and accelerates convolution by dynamically skipping unimportant input and output channels. FBS-augmented networks are trained with conventional stochastic gradient descent, making it readily available for many state-of-the-art CNNs. We compare FBS to a range of existing channel pruning and dynamic execution schemes and demonstrate large improvements on ImageNet classification. Experiments show that FBS can respectively provide $5\times$ and $2\times$ savings in compute on VGG-16 and ResNet-18, both with less than $0.6\%$ top-5 accuracy loss.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Signal Processing

Citations in the corpus

Dynamic Neural Networks: A Survey (2021, background): “to decide: 1) which channels need to be activated [15], [85], [86], [87], [88] width, 2) which layers need to be skipped [47], [48], [92], [93], 3) which paths should be selected in a SuperNet [107], or 4) what locations of the input should be allocated computations [146], [147], [148], [182].”
DISCO: Dynamic and Invariant Sensitive Channel Obfuscation for deep neural networks (2020, methodology): “Since different convolutional filters are known to activate differently [45], the dynamic channel pruning in DISCO enables more personalized identification of sensitive channels for each input resulting in better privacyutility trade-offs.”
CAP: Context-Aware Pruning for Semantic Segmentation (2021, neutral): “Unlike the traditional compression, more recent conditional computing based methods [2, 9] selectively switch the channels on/off based on the runtime activation.”
Batch-shaping for learning conditional channel gated networks (2019, background): “Dynamic Channel Pruning (Gao et al., 2018) turns individual features on or off similar to our approach, but they choose the top-k features instead, akin to Outrageously large neural networks (Shazeer et al.”
Continual Object Detection via Prototypical Task Correlation Guided Gating Mechanism (2022, positive): “It has been generally used to choose over different channels in dynamic pruning methods [4,8,10,13,19,21,29] and dynamic slimmable network [25].”
Learning Instance-wise Sparsity for Accelerating Deep Models (2019, methodology): “For example, [Gao et al., 2018; Lin et al., 2017; Odena et al., 2017; Hua et al., 2018] utilized attention and gate layers to evaluate each channel and discard some of them with subtle importances during the inference phrase.”
Dynamic Slimmable Network (2021, positive): “In previous dynamic networks [63, 64, 46, 68, 4, 30, 44, 29, 14, 9], the dynamic routing agent and the main network are jointly trained, analogous to jointly optimized NAS methods [48, 7, 66].”
Boosting the Performance of CNN Accelerators with Dynamic Fine-Grained Channel Gating (2019, result): “Table 4(a) and 4(b) compare CGNet to prior arts [5, 9, 24] on AlexNet and ResNet-18 for ImageNet.”
Channel Gating Neural Networks (2018, methodology): “Instead of bypassing residual unit, dynamic channel pruning [8] generates decisions to skip the computation for a subset of output channels.”
Fully Dynamic Inference With Deep Neural Networks (2020, background): “Within these layers, the contribution of individual channels is highly input-dependent [34].”
FrameExit: Conditional Early Exiting for Efficient Video Recognition (2021, background): “GaterNet [6], dynamic channel pruned networks [16], batch-shaped channel-gated networks [2], and Conditional channel gated networks for continual learning [1] perform gating in a more fine-grained fashion by turning individual channels of a layer on or off conditioned on the input.”
S2DNAS: Transforming Static CNN Model for Dynamic Inference via Neural Architecture Search (2019, methodology): “For example, well-designed gatefunctions were proposed as the controller to select a subset of channels or pixels for the subsequent computation of the convolution layer [7, 5, 15].”
Structured Pruning for Deep Convolutional Neural Networks: A Survey (2023, methodology): “In addition to using the next layer’s activation maps, Runtime Neural Pruning (RNP) [123] and Feature Boosting and Suppression (FBS) [124] utilize the layer l − 1 ’s (previous layer) activation maps to guide the pruning of the current layer.”
Dynamic Channel and Layer Gating in Convolutional Neural Networks (2020, methodology): “Gao et al. [6] proposed using Feature Boosting and Suppression (FBS) method.”
AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognition (2021, methodology): “If the policy only chooses from "skip" and "keep", then it becomes dynamic pruning methods (Gao et al., 2018; Hua et al., 2019).”
Pay Attention to Features, Transfer Learn Faster CNNs (2020, methodology): “Instead of removing neurons permanently from the network, FBS learns to dynamically prune unimportant channels, depending on the current input image.”
Batch-Shaped Channel Gated Networks (2019, background): “There are several works in recent literature that successfully learn conditional architectures, for example ConvNet-AIG [38] and Gao et al. [9].”
A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations (2023, neutral): “Gao et al. [148] propose Feature Boosting and Suppression ( FBS ) to predict the saliency of channels and skip those with less contribution to the classiﬁcation results at run-time.”
Dynamic Convolutions: Exploiting Spatial Sparsity for Faster Inference (2019, background): “Such practice is often called gating [11, 22], and can be applied at the level of convolutional layers [46, 48, 51], channels [3, 13, 30] or other elements in the network.”