Compressing Neural Networks using the Variational Information Bottleneck

By Bin Dai, Chen Zhu, David Paul Wipf. 2018

196 citations recorded by the source corpus.

Neural networks can be compressed to reduce memory and computational requirements, or to increase accuracy by facilitating the use of a larger base architecture. In this paper we focus on pruning individual neurons, which can simultaneously trim model size, FLOPs, and run-time memory. To improve upon the performance of existing compression algorithms we utilize the information bottleneck principle instantiated via a tractable variational bound. Minimization of this information theoretic bound reduces the redundancy between adjacent layers by aggregating useful information into a subset of neurons that can be preserved. In contrast, the activations of disposable neurons are shut off via an attractive form of sparse regularization that emerges naturally from this framework, providing tangible advantages over traditional sparsity penalties without contributing additional tuning parameters to the energy landscape. We demonstrate state-of-the-art compression rates across an array of datasets and network architectures.

Subfields: Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Efficiency & Compression

Citations in the corpus

The State of Sparsity in Deep Neural Networks (2019, methodology): “Many techniques grounded in Bayesian statistics and information theory have been proposed (Dai et al., 2018; Molchanov et al., 2017; Louizos et al., 2017b;a; Ullrich et al., 2017).”
Controlled Sparsity via Constrained Optimization or: How I Learned to Stop Tuning Penalties and Love Constraints (2022, methodology): “Structured pruning methods [6, 37, 31, 29], remove entire neurons/channels rather than individual parameters.”
Layer-compensated Pruning for Resource-constrained Convolutional Neural Networks (2018, methodology): “…that among the prior methods that we are comparing, ﬁve of them (Molchanov et al., 2017; Gordon et al., 2018; He et al., 2018b; Louizos et al., 2017; Dai et al., 2018) target the layer scheduling problem directly or indirectly while others focus on the complementary problems such as the ranking…”
Rigging the Lottery: Making All Tickets Winners (2019, methodology): “There are also structured pruning methods which attempt to remove channels or neurons so that the resulting network is dense and can be accelerated easily (Dai et al., 2018; Nek-lyudov et al., 2017; Christos Louizos, 2018).”
Batch-Shaped Channel Gated Networks (2019, methodology): “Similarly, full channels can be removed during training as in VIBnets [6], Bayesian Compression [22] and L0-regularization [23].”
Layer-adaptive Sparsity for the Magnitude-based Pruning (2020, methodology): “Other approaches include on-the-ﬂy 1 regularization (Louizos et al., 2018; Xiao et al., 2019), Bayesian methods (Molchanov et al., 2017; Louizos et al., 2017; Dai et al., 2018), and reinforcement learning (Lin et al., 2017).”
NETWORK COMPRESSION USING CORRELATION ANALYSIS OF LAYER RESPONSES (2018, methodology): “Recent work can be divided into two sub-families: sparse pruning (Han et al., 2016b; Srinivas & Babu, 2015; Wen et al., 2016; Tung & Mori, 2018; Aghasi et al., 2017; Carreira-Perpinan & Idelbayev, 2018; Zhang et al., 2018; Dai et al., 2018), where individual neurons are removed, and structured pruni”
Caveats for information bottleneck in deterministic scenarios (2018, unknown): “…the practical issue of estimating MI between neural network layers, an active area of recent research (Belghazi et al., 2018; Goldfeld et al., 2018; Dai et al., 2018; Gabrié et al., 2018), though our empirical experiments in Section 7 rely on the estimator proposed in (Kolchinsky & Tracey, 2017;…”
Fast Sparse ConvNets (2019, unknown): “Due to the difficultly of accelerating sparse computation, channel pruning approaches have been preferred [10, 5, 29, 28, 42, 13].”
Structured Pruning of Neural Networks With Budget-Aware Regularization (2018, methodology): “Among the structured pruning methods, those that implement a sparsity learning (SL) framework have shown to be effective as pruning and training are done simultaneously [1, 17, 21, 22, 24, 27].”
Structured Pruning for Deep Convolutional Neural Networks: A Survey (2023, methodology): “This structured pruning is based on the variational information bottleneck [104], and mixed-bit precision quantization is extended to power-of-two bit-restricted quantization.”
Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training (2021, methodology): “By relaxing the constraint of the fixed parameter count, Dai et al. (2019; 2018b) proposed a Grow-and-Prune strategy based on gradient-based growth and magnitude-based pruning to yield an accurate, yet very compact sparse network.”
Taxonomy and Evaluation of Structured Compression of Convolutional Neural Networks (2019, methodology): “(2017); Dai et al. (2018), and a recent survey is provided in Gale et al. (2019). Efficient architecture design.”
Nonparametric Bayesian Deep Networks with Local Competition (2018, unknown): “86 3/18 23/23/23/23 VIBNet (Dai et al., 2018) 1.”
Batch-shaping for learning conditional channel gated networks (2019, background): “Similarly, full channels can be removed during training as in VIBnets (Dai et al., 2018), Bayesian Compression (Louizos et al.”
Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communications (2022, unknown): “The IB problem is connected to multiple source-coding problems including source coding with logarithmic loss distortion [82], information combining [90], [91], common reconstruction [92], the Wyner-Ahlswede-Korner problem [93], [94], the efficiency of investment information [95]; to communications a”
Differentiable Joint Pruning and Quantization for Hardware Efficiency (2020, methodology): “59 VIBNet [4]+fixed quant.”