Loss-aware Binarization of Deep Networks

By Lu Hou, Quanming Yao, James T. Kwok. 2016

229 citations recorded by the source corpus.

Deep neural network models, though very powerful and highly successful, are computationally expensive in terms of space and time. Recently, there have been a number of attempts on binarizing the network weights and activations. This greatly reduces the network size, and replaces the underlying multiplications to additions or even XNOR bit operations. However, existing binarization schemes are based on simple matrix approximation and ignore the effect of binarization on the loss. In this paper, we propose a proximal Newton algorithm with diagonal Hessian approximation that directly minimizes the loss w.r.t. the binarized weights. The underlying proximal step has an efficient closed-form solution, and the second-order information can be efficiently obtained from the second moments already computed by the Adam optimizer. Experiments on both feedforward and recurrent networks show that the proposed loss-aware binarization algorithm outperforms existing binarization schemes, and is also more robust for wide and deep networks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neurology, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Neurology

In Model Architectures

In Optimization & Training

Citations in the corpus

The case for 4-bit precision: k-bit Inference Scaling Laws (2022, background): “…centering (Krishnamoorthi, 2018; Jacob et al., 2017), learned data types that are found through clustering (Gong et al., 2014; Han et al., 2015; Choi et al., 2016; Park et al., 2017), or direct codebook optimization (Rastegari et al., 2016; Hou et al., 2016; Leng et al., 2018; Zhang et al., 2018).”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “Important works here are loss-aware binarization and ternarization [97, 98] that directly minimize the loss with respect to the binarized/ternatized weights.”
TernaryBERT: Distillation-aware Ultra-low Bit BERT (2020, background): “…(Rastegari et al., 2016; Li et al., 2016) aims at keeping the quantized weights close to the full-precision weights, while loss-aware based quantization (Hou et al., 2017; Hou and Kwok, 2018; Leng et al., 2018) directly optimizes for the quantized weights that minimize the training loss.”
BinaryBERT: Pushing the Limit of BERT Quantization (2020, methodology): “Aside from BWN, we compare with LAB (Hou et al., 2017) and BiReal (Liu et al.”
Binary Neural Networks: A Survey (2020, background): “Improve Network Loss Function LAB [83] 1/32 VGG-Small 89.”
Loss aware post-training quantization (2019, methodology): “For example, Hou et al. (2016); Hou and Kwok (2018) have shown that we can utilize the Hessian of the loss to improve quantization of the CNNs.”
A comprehensive survey on model compression and acceleration (2020, methodology): “SqueezeNet, developed by Iandola et al. (2017), is a small CNN based architecture.”
Pruning and Quantization for Deep Neural Network Acceleration: A Survey (2021, methodology): “To address this issue, Hou [98] formulated the binarization effect on the network loss as an optimization problem which was solved by a proximal Newton’s algorithm with diagonal Hessian approximation that directly minimizes the loss with respect to the binary weights.”
MetaQuant: Learning to Quantize by Learning to Penetrate Non-differentiable Quantization (2019, methodology): ““Non-STE” methods are referred to as learning without STE by directly working on full-precision weights with a regularizer to obtain feasible quantization ([2]) or weights projection using proximal gradient methods ([10], [5]).”
Explicit Loss-Error-Aware Quantization for Low-Bit Deep Neural Networks (2018, methodology): “Numerous methods, including but not limited to [4] [23] [28] [45] [44] [19] [15] [14], are proposed to train binary or ternary neural networks.”
Efficient Neural Architecture Search via Proximal Iterations (2019, background): “Besides, while (3) empirically leads to better performance than (2) in binary networks (Courbariaux, Bengio, and David 2015; Hou, Yao, and Kwok 2017; Bai, Wang, and Liberty 2018), lazy-update (11) will not success here neither.”
A Survey on Methods and Theories of Quantized Neural Networks (2018, background): “More recently, [Lu Hou, 2018] extended [Hou et al., 2016] to lossaware ternarization and m-bit quantization.”
Model Compression and Acceleration for Deep Neural Networks: The Principles, Progress, and Challenges (2018, methodology): “To address this issue, the work in [17] proposed a proximal Newton algorithm with diagonal Hessian approximation that directly minimizes the loss with respect to the binary”
Compression of Generative Pre-trained Language Models via Quantization (2022, methodology): “Compared with PACT, LSQ and LAQ, our method not only aligns the token representations between the quantized and full-precision networks ( i.e. , diagonal boxes), but also captures the dependencies among different tokens (non-diagonal boxes).”
Training Compact Neural Networks with Binary Weights and Low Precision Activations (2018, methodology): “With more sophisticated quantization methods [13, 12] and optimization [42, 41], we expect that our performance may be further improved for more practical applications.”
Forward and Backward Information Retention for Accurate Binary Neural Networks (2019, methodology): “Table 3 lists the performance using different methods on CIFAR-10, including RAD [13] over ResNet-18 (based on [31]), DoReFa-Net [61], LQ-Net [59], DSQ [18] over ResNet-20 (based on [3]) and BNN [27], LAB [25], RAD, XNOR-Net [44] over VGG-Small.”
A Survey of Model Compression and Acceleration for Deep Neural Networks (2017, methodology): “To address this issue, the work in [17] proposed a proximal Newton algorithm with diagonal Hessian approximation that directly minimizes the loss with respect to the binary”