Domain-Adaptive Deep Network Compression

By Marc Masana, Joost van de Weijer, Luis Herranz, Andrew D. Bagdanov, José Manuel Álvarez. 2017

64 citations recorded by the source corpus.

Deep Neural Networks trained on large datasets can be easily transferred to new domains with far fewer labeled examples by a process called fine-tuning. This has the advantage that representations learned in the large source domain can be exploited on smaller target domains. However, networks designed to be optimal for the source task are often prohibitively large for the target task. In this work we address the compression of networks after domain transfer. We focus on compression algorithms based on low-rank matrix decomposition. Existing methods base compression solely on learned network weights and ignore the statistics of network activations. We show that domain transfer leads to large shifts in network activations and that it is desirable to take this into account when compressing. We demonstrate that considering activation statistics when compressing weights leads to a rank-constrained regression problem with a closed-form solution. Because our method takes into account the target domain, it can more optimally remove the redundancy in the weights. Experiments show that our Domain Adaptive Low Rank (DALR) method significantly outperforms existing low-rank compression techniques. With our approach, the fc6 layer of VGG19 can be compressed more than 4x more than using truncated SVD alone – with only a minor or no loss in accuracy. When applied to domain-transferred networks it allows for compression down to only 5-20% of the original number of parameters with only a minor drop in performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Efficiency & Compression

Citations in the corpus

Efficient Structured Pruning and Architecture Searching for Group Convolution (2018, background): “, L1, L2 norms [27, 5, 20, 26], firstorder [29, 3] or second-order [19, 7, 2] gradients, average percentage of zero [26, 11], singular values [32, 28].”
AMC: AutoML for Model Compression and Acceleration on Mobile Devices (2018, background): “Tensor factorization [30, 15, 27, 35] decomposes weights into light-weight pieces, for example [51, 11, 14] proposed to accelerate the fully connected layers with truncated SVD; Jaderberg et al .”
Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgetting (2018, methodology): “The use of layer decomposition with SVD was also investigated in the context of network compression [20], [21].”
Ternary Feature Masks: continual learning without any forgetting (2020, background): “Feature usage on the new tasks look very unbalanced in comparison to learning larger tasks [37].”
Extreme Network Compression via Filter Group Approximation (2018, methodology): “[23] proposed a domain adaptive low-rank decomposition method, which took the activations’ statistics of the new domain into account.”
NETWORK COMPRESSION USING CORRELATION ANALYSIS OF LAYER RESPONSES (2018, methodology): “…present in convolution layers by replacing the original tensors with a sequence of smaller or sparser counterparts that produce similar responses (Denton et al., 2014; Lebedev et al., 2014; Jaderberg et al., 2014; Cheng et al., 2016; Masana et al., 2017; Yu et al., 2017; Peng et al., 2018).”
Ternary Feature Masks: zero-forgetting for task-incremental learning (2021, background): “Feature usage on the new tasks look very unbalanced in comparison to learning larger tasks [30].”
Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, methodology): “Masana et al. [83] used SVD to decompose the product of the input and the weight matrix.”
ADC: Automated Deep Compression and Acceleration with Reinforcement Learning (2018, unknown): “Tensor factorization [34, 18, 31, 41] decomposed weights into several pieces.”
Finding Sparse Structures for Domain Specific Neural Machine Translation (2021, unknown): “Our approach is also inspired by many studies of sparse networks (Frankle and Carbin 2019; Zhu and Gupta 2018; Liu et al. 2019; Masana et al. 2017).”