On Compressing Deep Models by Low Rank and Sparse Decomposition

By Xiyu Yu, Tongliang Liu, Xinchao Wang, Dacheng Tao. 2017

482 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Xinchao Wang
3. Christopher R'e
4. Yunhe Wang
5. Cho-Jui Hsieh
6. Percy Liang
7. Tri Dao
8. Quoc V. Le

In Computational Mechanics

1. Christopher R'e
2. Xinchao Wang
3. Tri Dao
4. Taiji Suzuki
5. Mingli Song
6. Atri Rudra
7. Cho-Jui Hsieh
8. Dacheng Tao

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Luc Van Gool
3. Xinchao Wang
4. Jiashi Feng
5. Ming-Hsuan Yang
6. Yi Yang
7. Xiaogang Wang
8. Qi Tian

Citations in the corpus

• LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks (2018, methodology): “Representative techniques include quantization [6, 34, 29, 54, 52, 21, 39, 53, 22, 3, 55, 9, 18], pruning [13, 12, 17, 36], low-rank decomposition [7, 8, 24, 27, 38, 51, 49], hashing [4], and deliberate architecture design [23, 19, 50].”
• Distilling Knowledge From Graph Convolutional Networks (2020, background): “To this end, knowledge distillation has been pro-a posed, which is one of the main streams of model compression [14, 2, 38, 46].”
• Progressive Network Grafting for Few-Shot Knowledge Distillation (2020, unknown): “Resuing pre-trained networks has recently attracted attentions from researchers in the field (Chen et al. 2020; Yu et al. 2017).”
• Provable Filter Pruning for Efficient Neural Networks (2019, background): “These approaches range from those inspired by classical tensor decompositions (Yu et al., 2017b; Jaderberg et al., 2014; Denton et al., 2014), and random projections and hashing (Arora et al., 2018; Ullrich et al., 2017; Chen et al., 2015; Weinberger et al., 2009; Shi et al., 2009) that compress a…”
• NETWORK COMPRESSION USING CORRELATION ANALYSIS OF LAYER RESPONSES (2018, background): “…present in convolution layers by replacing the original tensors with a sequence of smaller or sparser counterparts that produce similar responses (Denton et al., 2014; Lebedev et al., 2014; Jaderberg et al., 2014; Cheng et al., 2016; Masana et al., 2017; Yu et al., 2017; Peng et al., 2018).”
• Amalgamating Filtered Knowledge: Learning Task-customized Student from Multi-task Teachers (2019, background): “First proposed in [Hinton et al., 2015], knowledge distillation aims at training a student model of a compact size by learning from a larger teacher model or a set of teachers handling the same task, and thus finds its important application in deep model compression [Yu et al., 2017].”
• Kaleidoscope: An Efficient, Learnable Representation For All Structured Linear Maps (2020, methodology): “…such as the discrete Fourier transform (DFT) and Hadamard transform used in signal processing (Cooley et al., 1969), convolutions for image, language, and speech modeling (Gu et al., 2018), and low-rank and sparse matrices for efficient storage and inference on edge devices (Yu et al., 2017).”
• Decomposable-Net: Scalable Low-Rank Compression for Neural Networks (2019, methodology): “ into K w 1 and 1 K h. Denton et al. (2014); Lebedev et al. (2015); Kim et al. (2016) proposed methods that use tensor factorization without rearranging the convolutional tensor into the matrix form. Yu et al. (2017) further improved the compression rate by incorporating sparseness into the low-rank”
• Collaboration by Competition: Self-coordinated Knowledge Amalgamation for Multi-talent Student Learning (2020, background): “In this way, the large teacher can transfer knowledge to the student with smaller model size, which is widely applied to model compression [32].”
• Medical Image Segmentation Review: The Success of U-Net (2022, methodology): “Deep model compression techniques such as pruning [201], quantization [201], lowrank approximation [202], knowledge distillation [203], and neural architecture search [204] to new a few, is an area”
• Triple Wins: Boosting Accuracy, Robustness and Efficiency Together by Enabling Input-Adaptive Inference (2020, methodology): “Layer factorization (Tai et al., 2016; Yu et al., 2017), quantization (Wu et al., 2016), model distillation (Wang et al., 2018c) and weight sharing (Wu et al., 2018) have also been respectively found effective.”
• A Weakly Supervised Fine Label Classifier Enhanced by Coarse Supervision (2019, background): “The low rank and sparse representation of the high dimensional data is based on the fact that a correlation often exists among the data which belong to the same class such that a low rank subspace captures the global and smooth structures, while a sparse subspace captures the local structures and fi”
• Variational Convolutional Neural Network Pruning (2019, unknown): “These approaches include tensor factorization [35, 48], net-”
• Data-Free Knowledge Amalgamation via Group-Stack Dual-GAN (2020, background): “First proposed in [13], knowledge distillation aims at training a student model of a compact size by learning from a larger teacher model or a set of teachers handling the same task, and thus finds its important application in deep model compression [37].”
• Towards Evolutionary Compression (2018, background): “To overcome this conflict between reduced hardware configurations and the higher resource demands of CNNs, several attempts have beenmade to compress and speed upwell-trained CNNmodels by refining their parameters including weight pruning [2, 13, 14, 49], quantization and binarization [1, 5, 6, 20],”
• Data-Dependent Coresets for Compressing Neural Networks with Applications to Generalization Bounds (2018, background): “Similar works entail the use of low-rank tensor decomposition approaches applicable both during and after training [26, 28, 45, 25, 4, 52].”