Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMM

By Cong Leng, Hao Li, Shenghuo Zhu, Rong Jin. 2017

305 citations recorded by the source corpus.

Although deep learning models are highly effective for various learning tasks, their high computational costs prohibit the deployment to scenarios where either memory or computational resources are limited. In this paper, we focus on compressing and accelerating deep models with network weights represented by very small numbers of bits, referred to as extremely low bit neural network. We model this problem as a discretely constrained optimization problem. Borrowing the idea from Alternating Direction Method of Multipliers (ADMM), we decouple the continuous parameters from the discrete constraints of network, and cast the original hard problem into several subproblems. We propose to solve these subproblems using extragradient and iterative quantization algorithms that lead to considerably faster convergency compared to conventional optimization methods. Extensive experiments on image recognition and object detection verify that the proposed algorithm is more effective than state-of-the-art approaches when coming to extremely low bit neural network.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

The case for 4-bit precision: k-bit Inference Scaling Laws (2022, background): “…centering (Krishnamoorthi, 2018; Jacob et al., 2017), learned data types that are found through clustering (Gong et al., 2014; Han et al., 2015; Choi et al., 2016; Park et al., 2017), or direct codebook optimization (Rastegari et al., 2016; Hou et al., 2016; Leng et al., 2018; Zhang et al., 2018).”
Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search (2018, background): “Early works such as Han et al. (2015); Zhu et al. (2016); Leng et al. (2017) mainly focus on quantizing neural network weights while still using 32-bit activations.”
Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference (2017, methodology): “2: ResNet on ImageNet: Accuracy under various quantization schemes, including binary weight networks (BWN [21, 15]), ternary weight networks (TWN [21, 22]), incremental network quantization (INQ [33]) and fine-grained quantization (FGQ [26])”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “While STE is the mainstream approach [218, 280], other approaches have also been explored in the literature [2, 25, 31, 59, 142, 160].”
TernaryBERT: Distillation-aware Ultra-low Bit BERT (2020, background): “…(Rastegari et al., 2016; Li et al., 2016) aims at keeping the quantized weights close to the full-precision weights, while loss-aware based quantization (Hou et al., 2017; Hou and Kwok, 2018; Leng et al., 2018) directly optimizes for the quantized weights that minimize the training loss.”
A Survey on Methods and Theories of Quantized Neural Networks (2018, methodology): “In [Leng et al., 2017], the authors formulated the quantization problem as a discretely constraint non-convex optimization problem and used the idea of Alternating Direction Method of Multipliers (ADMM) to decouple the continuous variables from the discrete constraints.”
Pre-Defined Sparsity for Low-Complexity Convolutional Neural Networks (2020, background): “Previous work has focused on accelerating inference and proposed model pruning [16], [17], [18], [19], [20] and quantization [21], [22], [23], [24], [25], [26] to reduce the number of nonzero parameters.”
Dynamic Network Quantization for Efficient Video Inference (2021, unknown): “Early works such as [16, 31, 69] mainly focus on quantizing weights while using 32-bit activations.”
Binary Neural Networks: A Survey (2020, methodology): “applied BNNs to different tasks, and evaluated their method on convolutional neural networks for image classification and object detection, and recurrent neural networks for language model [121].”
Quantization Networks (2019, methodology): “As in [27, 21, 20], we use AlexNet coupled with batch normalization [14] layers.”
Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation (2020, methodology): “[31] used the Alternating Direction Method of Multipliers (ADMM) as an alternative to STE when training quantized model.”
A review of convolutional neural network architectures and their optimizations (2022, background): “1 3 parameters, thereby transforming the original problem into several sub-problems (Leng et al. 2018).”
Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, methodology): “Whereas, on larger data sets such as ImageNet, the optimization methods often perform better [162], [165].”
AdaBits: Neural Network Quantization With Adaptive Bit-Widths (2019, methodology): “Initially, uniform precision quantization is adopted inside the whole network, where all layers share the same bit-width [17, 19, 28, 31, 32, 33, 46, 52].”
Differentiable Feature Aggregation Search for Knowledge Distillation (2020, background): “Therefore, the community has raised increasing attention on model compression approaches such as model pruning [7,21,34], model quantization [14,19,23] and knowledge distillation [13,30,33,37,38].”
A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers (2018, background): “In a convolutional layer the weights are organized in a four-dimensional tensor and in a fully-connected layer they are organized in a two-dimensional matrix [20].”
CGaP: Continuous Growth and Pruning for Efficient Deep Learning (2019, background): “[Denton et al., 2014; Leng et al., 2018] compressed each convolutional layer by finding an appropriate low-rank approximation.”
Pruning and Quantization for Deep Neural Network Acceleration: A Survey (2021, background): “Li [137] and Leng [135] showed that for ternary weights (−1, 0, and + 1), ternary weight networks (TWN) result in a slight accuracy loss and much higher accuracy than binary weights models.”
MetaQuant: Learning to Quantize by Learning to Penetrate Non-differentiable Quantization (2019, methodology): ““Non-STE” methods are referred to as learning without STE by directly working on full-precision weights with a regularizer to obtain feasible quantization ([2]) or weights projection using proximal gradient methods ([10], [5]).”