Ternary Neural Networks with Fine-Grained Quantization

By Naveen Mellempudi, Abhisek Kundu, Dheevatsa Mudigere, Dipankar Das, Bharat Kaul, Pradeep K. Dubey. 2017

112 citations recorded by the source corpus.

We propose a novel fine-grained quantization (FGQ) method to ternarize pre-trained full precision models, while also constraining activations to 8 and 4-bits. Using this method, we demonstrate a minimal loss in classification accuracy on state-of-the-art topologies without additional training. We provide an improved theoretical formulation that forms the basis for a higher quality solution using FGQ. Our method involves ternarizing the original weight tensor in groups of $N$ weights. Using $N=4$, we achieve Top-1 accuracy within $3.7\%$ and $4.2\%$ of the baseline full precision result for Resnet-101 and Resnet-50 respectively, while eliminating $75\%$ of all multiplications. These results enable a full 8/4-bit inference pipeline, with best-reported accuracy using ternary weights on ImageNet dataset, with a potential of $9\times$ improvement in performance. Also, for smaller networks like AlexNet, FGQ achieves state-of-the-art results. We further study the impact of group size on both performance and accuracy. With a group size of $N=64$, we eliminate $\approx99\%$ of the multiplications; however, this introduces a noticeable drop in accuracy, which necessitates fine tuning the parameters at lower precision. We address this by fine-tuning Resnet-50 with 8-bit activations and ternary weights at $N=64$, improving the Top-1 accuracy to within $4\%$ of the full precision result with $<30\%$ additional training overhead. Our final quantized model can run on a full 8-bit compute pipeline using 2-bit weights and has the potential of up to $15\times$ improvement in performance compared to baseline full-precision models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Model compression via distillation and quantization (2018, background): “…is the work on training quantized neural networks, e.g. Courbariaux et al. (2015); Rastegari et al. (2016); Hubara et al. (2016); Wu et al. (2016a); Mellempudi et al. (2017); Ott et al. (2016); Zhu et al. (2016), which showed that neural networks can converge to good task solutions even when…”
Bayesian Compression for Deep Learning (2017, methodology): “As an extreme example, [13, 48, 57, 76] and [11] trained NN to use only binary or tertiary weights with floating point gradients.”
Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference (2017, methodology): “This methodology, embraced by approaches such as Ternary weight networks (TWN [22]), Binary Neural Networks (BNN [14]), XNOR-net [27], and more [8, 21, 26, 33, 34, 35], is the focus of our investigation.”
Efficient Contextualized Representation: Language Model Pruning for Sequence Labeling (2018, methodology): “Typically, NN compression methods require the re-training of the whole model (Mellempudi et al., 2017).”
Pruning and Quantization for Deep Neural Network Acceleration: A Survey (2021, background): “With re-training, FGQ could significantly improve top-1 accuracy improving from 48% on non-trained to 71.1% top-1 on ResNet-50.”
A Survey on Methods and Theories of Quantized Neural Networks (2018, background): “In [Mellempudi et al., 2017], the authors introduced multiple scaling factors into ternary network to account for the unsymmetry between positive and negative weights.”
Loss aware post-training quantization (2019, methodology): “Many successful methods of post-training quantization perform finer parameter assignment, such as group-wise (Mellempudi et al. 2017), channel-wise (Banner et al. 2018), pixel-wise (Fraser et al. 2018), or filter-wise (Kravchik et al. 2019) quantization, requiring special hardware support and…”
Inducing and Exploiting Activation Sparsity for Fast Neural Network Inference (2020, background): “…be examined in the broader context of model compression methods, which is an extremely active research area, e.g. (Wu et al., 2016; Zhu et al., 2016; Mishra et al., 2017; Mellempudi et al., 2017; Zhang et al., 2017; Park et al., 2016b; Han et al., 2016; Polino et al., 2018; Frankle & Carbin, 2018).”
Low-bit Quantization of Neural Networks for Efficient Inference (2019, background): “For example, with the efﬁ-cient group-wise [1, 27], channel-wise or pixel-wise quantization settings [8], the dot product must be split into several dot products to be accumulated together.”
SYQ: Learning Symmetric Quantization for Efficient Deep Neural Networks (2018, background): “More fine-grained approaches of quantization have effectively clustered weights or grouped filters together and quantize differently based on their statistical distributions [6], [21] .”
A Study of BFLOAT16 for Deep Learning Training (2019, background): “Inference has witnessed a proliferation of mixed precision compute [19, 33, 27, 25] where different operations execute at different precision, all the way from binary/ternary operands to 16b ﬂoating point.”
Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy (2017, methodology): “Works like Binary connect (BC) (Courbariaux et al., 2015), Ternary-weight networks (TWN) (Li & Liu, 2016), ﬁne-grained ternary quantization (Mellempudi et al., 2017) and INQ (Zhou et al., 2017) target precision reduction of network weights.”
WRPN: Wide Reduced-Precision Networks (2017, methodology): “Works like Binary connect (BC) [5], Ternary-weight networks (TWN) [12], ﬁne-grained ternary quantization [14] and INQ [25] target reducing the precision of network weights while still using full-precision activations.”
Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation (2020, background): “Much of the earlier research in this area focused on very low bit quantization [7, 13, 59], all the way down to ternary (2-bit) [60, 34] and binary weights [8] and activations [45, 18].”
AdaBits: Neural Network Quantization With Adaptive Bit-Widths (2019, methodology): “Initially, uniform precision quantization is adopted inside the whole network, where all layers share the same bit-width [20, 40, 28, 25, 26, 37, 27, 19].”
Extreme Compression for Pre-trained Transformers Made Simple and Efficient (2022, methodology): “Popular quantization methods include post-training quantization [45, 33, 29], quantization-aware training (QAT) [23, 4, 31, 22], and their variations [44, 24, 12].”
PACT: Parameterized Clipping Activation for Quantized Neural Networks (2018, methodology): “• Fine-grained Quantization (FGQ, Mellempudi et al. (2017)): A direct quantization scheme (i.e., little re-training needed) based on ﬁne-grained grouping (i.e., within a small subset of ﬁlter maps).”