Channel Gating Neural Networks

By Weizhe Hua, Christopher De Sa, Zhiru Zhang, G. Suh. 2018

207 citations recorded by the source corpus.

This paper introduces channel gating, a dynamic, fine-grained, and hardware-efficient pruning scheme to reduce the computation cost for convolutional neural networks (CNNs). Channel gating identifies regions in the features that contribute less to the classification result, and skips the computation on a subset of the input channels for these ineffective regions. Unlike static network pruning, channel gating optimizes CNN inference at run-time by exploiting input-specific characteristics, which allows substantially reducing the compute cost with almost no accuracy loss. We experimentally show that applying channel gating in state-of-the-art networks achieves 2.7-8.0$\times$ reduction in floating-point operations (FLOPs) and 2.0-4.4$\times$ reduction in off-chip memory accesses with a minimal accuracy loss on CIFAR-10. Combining our method with knowledge distillation reduces the compute cost of ResNet-18 by 2.6$\times$ without accuracy drop on ImageNet. We further demonstrate that channel gating can be realized in hardware efficiently. Our approach exhibits sparsity patterns that are well-suited to dense systolic arrays with minimal additional hardware. We have designed an accelerator for channel gating networks, which can be implemented using either FPGAs or ASICs. Running a quantized ResNet-18 model for ImageNet, our accelerator achieves an encouraging speedup of 2.4$\times$ on average, with a theoretical FLOP reduction of 2.8$\times$.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Dynamic Neural Networks: A Survey (2021, methodology): “(CGNet) [84] first executes a subset of convolutional filters in every layer, and the remaining filters are only activated on strategically selected areas.”
A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations (2023, unknown): “[147]exploitinput-speciﬁccharacteristicsandpropose CGNets to predict unimportant regions by the partial sum of the output activation by performing convolution on a subset of input channels.”
Boosting the Performance of CNN Accelerators with Dynamic Fine-Grained Channel Gating (2019, methodology): “To further improve the accuracy of CGNets, channel grouping and shuffling can be used to “equalize” the importance of each channel [14].”
A Survey on Federated Learning for Resource-Constrained IoT Devices (2021, unknown): “DSG techniques ([181], [182]) identify spatial regions in the OFM that are deemed important and focus their attention only on those parts of the OFM.”
Dynamic Channel Pruning: Feature Boosting and Suppression (2018, background): “Bolukbasi et al. (2017) trained differentiable policy functions to implement early exits in a DNN. Hua et al. (2018) learned binary policies that decide whether partial or all input channels are used for convolution, but approximate the gradients of the non-differentiable policy functions with…”
Glance and Focus: a Dynamic Approach to Reducing Spatial Redundancy in Image Classification (2020, methodology): “The methods proposed in [19] and [62] skip the computation on some less important regions of feature maps.”
AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognition (2021, methodology): “If the policy only chooses from "skip" and "keep", then it becomes dynamic pruning methods (Gao et al., 2018; Hua et al., 2019).”
Learning Instance-wise Sparsity for Accelerating Deep Models (2019, methodology): “For example, [ Gao et al. , 2018; Lin et al. , 2017; Hua et al. , 2018 ] utilized attention and gate layers to evaluate each channel and discard some of them with subtle importances during the inference phrase.”
S2DNAS: Transforming Static CNN Model for Dynamic Inference via Neural Architecture Search (2019, methodology): “For example, well-designed gatefunctions were proposed as the controller to select a subset of channels or pixels for the subsequent computation of the convolution layer [7, 5, 15].”
Continual Object Detection via Prototypical Task Correlation Guided Gating Mechanism (2022, unknown): “It has been generally used to choose over different channels in dynamic pruning methods [4,8,10,13,19,21,29] and dynamic slimmable network [25].”
Sponge Examples: Energy-Latency Attacks on Neural Networks (2020, background): “On the algorithmic level, there is a recent surge of interest in using dynamic coarse-grained sparsity for accelerating GPU inference [52, 53].”
AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition (2021, background): “Channel gating network [21] identifies regions in the features that contribute less to the classification result, and skips the computation on a subset of the input channels for these ineffective regions.”
Dynamic Network Quantization for Efficient Video Inference (2021, unknown): “Representative meth-ods for image classiﬁcation, dynamically adjust network depth [13, 33, 59, 21, 63], width [66, 7, 20], perform routing [26, 33] or switch resolutions [62].”
Self-Adaptive Network Pruning (2019, methodology): “Channel Gating (CG) Hua et al. (2018) splits channels in each layer into two groups and the proportion of the first group is uniform for all layers.”
VA-RED2: Video Adaptive Redundancy Reduction (2021, methodology): “, 2015b) and a automatic channel pruning method (CGNet) (Hua et al., 2019) on Mini-Kinetics-200.”
Dynamic Slimmable Network (2021, unknown): “In previous dynamic networks [63, 64, 46, 68, 4, 30, 44, 29, 14, 9], the dynamic routing agent and the main network are jointly trained, analogous to jointly optimized NAS methods [48, 7, 66].”