CLIP-Q: Deep Network Compression Learning by In-parallel Pruning-Quantization

By Frederick Tung, Greg Mori. 2018

223 citations recorded by the source corpus.

Deep neural networks enable state-of-the-art accuracy on visual recognition tasks such as image classification and object detection. However, modern deep networks contain millions of learned weights; a more efficient utilization of computation resources would assist in a variety of deployment scenarios, from embedded platforms with resource constraints to computing clusters running ensembles of networks. In this paper, we combine network pruning and weight quantization in a single learning framework that performs pruning and quantization jointly, and in parallel with fine-tuning. This allows us to take advantage of the complementary nature of pruning and quantization and to recover from premature pruning errors, which is not possible with current two-stage approaches. Our proposed CLIP-Q method (Compression Learning by In-Parallel Pruning-Quantization) compresses AlexNet by 51-fold, GoogLeNet by 10-fold, and ResNet-50 by 15-fold, while preserving the uncompressed network accuracies on ImageNet.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Similarity-Preserving Knowledge Distillation (2019, methodology): “Weight pruning methods [11, 20, 22, 24, 33, 35, 39] remove unimportant weights from the network, sparsifying the network connectivity structure.”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “Some work in the literature has also explored non-uniform quantization [25, 38, 62, 74, 78, 98, 117, 123, 150, 156, 175, 184, 185, 230, 240, 248, 256, 258, 268, 275], where quantization steps as well as quantization levels are allowed to be non-uniformly spaced.”
Variational Convolutional Neural Network Pruning (2019, methodology): “These approaches include tensor factorization [35,48] work quantization [4, 40], and weight pruning [11, 32].”
Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks With Octave Convolution (2019, methodology): “Since OctConv mainly focuses on processing feature maps at multiple spatial frequencies and reducing their spatial redundancy, it is orthogonal and complementary to existing methods that focus on building better CNN topology [22, 41, 35, 33, 29], reducing channel-wise redundancy in convolutional fea”
Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks (2019, background): “Besides, there are other non-uniform methods like [30, 13, 3, 35, 37, 38, 44] which may need delicate hardware to get acceleration.”
Structured Pruning for Deep Convolutional Neural Networks: A Survey (2023, methodology): “In addition, the quantization process also includes clustering and parameter sharing [40], and quantization can be combined with pruning [1], [284], [285].”
Differentiable Joint Pruning and Quantization for Hardware Efficiency (2020, methodology): “evoort a ﬂexible trade-oﬀ between the two. For an overview of structured pruning methods, please refer to [16]. To jointly optimize pruning and quantization, two research works have been proposed [35][28]. Unfortunately, neither of them support activation quantization. Thus, the resulting networks c”
Adversarial Network Compression (2018, methodology): “ar to binary approaches, ternary weights (-1,0,1) have been employed as well [31]. II. Pruning Reducing the model size (memory and storage) is also the goal of pruning by removing network connections [5,51,60]. At the same time, it prevents over-tting. In [16], the unimportant connections of the net”
Meta Filter Pruning to Accelerate Deep Convolutional Neural Networks (2019, background): “[9, 8, 7, 30, 2, 36, 4] focus on pruning fine-grained weight of filters, which leads to unstructured sparsity in models.”
Constraint-Aware Deep Neural Network Compression (2018, methodology): “We used magnitude-based pruning [6, 8, 15, 17] followed by fine-tuning as our compression strategy Φ.”
Pruning Filter via Geometric Median for Deep Convolutional Neural Networks Acceleration (2018, background): “Many recent works [9, 8, 7, 26, 1, 12, 31] focus on pruning fine-grained weight of filters.”
NETWORK COMPRESSION USING CORRELATION ANALYSIS OF LAYER RESPONSES (2018, background): “Recent work can be divided into two sub-families: sparse pruning (Han et al., 2016b; Srinivas & Babu, 2015; Wen et al., 2016; Tung & Mori, 2018; Aghasi et al., 2017; Carreira-Perpinan & Idelbayev, 2018; Zhang et al., 2018; Dai et al., 2018), where individual neurons are removed, and structured…”