Compression-aware Training of Deep Networks

By José Manuel Álvarez, Mathieu Salzmann. 2017

182 citations recorded by the source corpus.

In recent years, great progress has been made in a variety of application domains thanks to the development of increasingly deeper neural networks. Unfortunately, the huge number of units of these networks makes them expensive both computationally and memory-wise. To overcome this, exploiting the fact that deep networks are over-parametrized, several compression strategies have been proposed. These methods, however, typically start from a network that has been trained in a standard manner, without considering such a future compression. In this paper, we propose to explicitly account for compression in the training process. To this end, we introduce a regularizer that encourages the parameter matrix of each layer to have low rank during training. We show that accounting for compression during training allows us to learn much more compact, yet at least as effective, models than state-of-the-art compression techniques.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Extreme Network Compression via Filter Group Approximation (2018, background): “[1] regularized parameter matrix to low-rank during training, such that it could be decomposed easily in the post-processing stage.”
Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification (2020, methodology): “On the other hand, Alvarez et al. [1] and Xu et al. [35] explicitly estimate and reduce the rank throughout the training by adding Nuclear Norm (deﬁned as the sum of all singular values) regularizer to the training objective.”
PruneTrain: Gradual Structured Pruning from Scratch for Faster Neural Network Training (2019, methodology): “ process a sparse CNN structure [6, 12, 13]. The closest prior work to ours reconfigures the CNN architecture exactly once during the training process, processing the smaller model from that point on [8]. However, the reconfiguration point is not known a priori, making applying this approach problem”
Separable Layers Enable Structured Efficient Linear Substitutions (2019, background): “…results on contemporary image classification problems for members of a number of classes of transformations: • Low-rank substitutions (Alvarez & Salzmann, 2016; Alvarez & Salzmann, 2017; Sainath et al., 2013; Jaderberg et al., 2014; Novikov et al., 2015; Kossaifi et al., 2017; Kim et al., 2015).”
Reborn Filters: Pruning Convolutional Neural Networks with Limited Data (2020, background): “Model compression serves as a remedy by investigating and further eliminating the redundancy in the networks (Alvarez and Salzmann 2017; You et al. 2017a).”
Accelerometer: Understanding Acceleration Opportunities for Data Center Overheads at Hyperscale (2020, background): “Additionally, Web, Feed1, Feed2, and Cache1 spend several cycles in compression and serialization (similar to Google’s services [63]); they can beneft from accelerating these common orchestration overheads [21, 27, 36, 47, 68, 106].”
Provable Filter Pruning for Efficient Neural Networks (2019, unknown): “…2009) that compress a pre-trained network, to those approaches that enable sparsity by embedding sparsity as an objective directly in the training process (Ioannou et al., 2015; Alvarez & Salzmann, 2017) or exploit tensor structure to induce sparsity (Choromanska et al., 2016; Zhao et al., 2017).”
Data-Independent Neural Pruning via Coresets (2019, methodology): “…of model compression heuristics have been successfully applied to different neural network models, such as Jacob et al. (2018); Han et al. (2015); Alvarez & Salzmann (2017), these approaches generally lack strong provable guarantees on the trade-off between the compression rate and the…”
FRILL: A Non-Semantic Speech Embedding for Mobile Devices (2020, unknown): “Similar to [18], we aim to learn the low-rank approximates during training as opposed to post-training.”
Data-Dependent Coresets for Compressing Neural Networks with Applications to Generalization Bounds (2018, background): “Similar works entail the use of low-rank tensor decomposition approaches applicable both during and after training [26, 28, 45, 25, 4, 52].”
Decomposable-Net: Scalable Low-Rank Compression for Neural Networks (2019, methodology): “(3) During learning, [Alvarez and Salzmann, 2017; Xu et al., 2020] utilized trace-norm regularization as a low-rank constraint.”
A Low Effort Approach to Structured CNN Design Using PCA (2018, methodology): “While this is not the ﬁrst work that uses PCA to analyze models [7], [9], the focus in this work is on a practical method of compressing pre-trained networks that does not involve multiple iterations of retraining.”
Mixed Dimension Embeddings with Application to Memory-Efficient Recommendation Systems (2019, background): “Online compression algorithms include quantization-aware training, gradual pruning, and periodic regularization [3, 16, 17, 35, 40].”
ExpandNets: Linear Over-parameterization to Train Compact Convolutional Networks (2018, background): “While compression is typically performed as a post-processing step, it has been shown that incorporating it during training could be beneficial [3, 4, 56, 57].”
Low-Rank Compression of Neural Nets: Learning the Rank of Each Layer (2020, methodology): “The nuclear norm was used to reduce the number of parameters in single- [16] and multi-layer networks [2].”
A comprehensive survey on model compression and acceleration (2020, methodology): “In addition, Alvarez and Salzmann (2017) also suggests the low-rank decomposition of the filters during training instead of a pre-trained network. To encourage the low-rank value for each layer matrix parameters, the authors introduce regularizer and to achieve more compression, regularizer is furth”