Ternary Weight Networks

By Fengfu Li. 2016

592 citations recorded by the source corpus.

We present a memory and computation efficient ternary weight networks (TWNs) - with weights constrained to +1, 0 and -1. The Euclidian distance between full (float or double) precision weights and the ternary weights along with a scaling factor is minimized in training stage. Besides, a threshold-based ternary function is optimized to get an approximated solution which can be fast and easily computed. TWNs have shown better expressive abilities than binary precision counterparts. Meanwhile, TWNs achieve up to 16× model compression rate and need fewer multiplications compared with the float32 precision counterparts. Extensive experiments on MNIST, CIFAR-10, and ImageNet datasets show that the TWNs achieve much better result than the Binary-Weight-Networks (BWNs) and the classification performance on MNIST and CIFAR-10 is very close to the full precision networks. We also verify our method on object detection task and show that TWNs significantly outperforms BWN by more than 10% mAP on PASCAL VOC dataset. The pytorch version of source code is available at: https://github.com/Thinklab-SJTU/twns.

Subfields: Computer Vision and Pattern Recognition, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Model Architectures

In Efficiency & Compression

Citations in the corpus

ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers (2022, background): “This analysis results also indicate why more expensive hidden-states knowledge distillation [2, 36] is used for ultra-low precision quantization to close the accuracy gap.”
Differentiable Model Compression via Pseudo Quantization Noise (2021, methodology): “Early network quantization methods focused on low-bitwidth networks such as BinaryNet (Courbariaux, Bengio, and David 2015; Courbariaux et al. 2016), XOR-Nets (Rastegari et al. 2016), or Ternary networks (Li, Zhang, and Liu 2016; Wu et al. 2018).”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “…problem: Furthermore, inspired by the observation that many learned weights are close to zero, there have been attempts to ternarize network by constraining the weights/activations with ternary values, e.g., +1, 0 and -1, thereby explicitly permitting the quantized values to be zero [143, 156].”
BinaryBERT: Pushing the Limit of BERT Quantization (2020, methodology): “We use loss-aware weight quantization (LAQ) (Hou and Kwok, 2018) for 8/4/3-bit weight quantization, TWN (Li et al., 2016) for weight ternarization and BWN (Hubara et al., 2016) for weight binarization.”
I-BERT: Integer-only BERT Quantization (2021, neutral): “These techniques can be broadly categorized into (1) model pruning [18, 31, 35, 38, 40, 67], (2) knowledge distillation [21, 39, 43, 49, 70], (3) efficient neural architecture design [23, 24, 37, 51, 57], (4) hardware and neural architecture co-design [16, 17, 22, 29, 64], and (5) quantization [5, 7”
Distribution Adaptive INT8 Quantization for Training CNNs (2021, unknown): “TWNs (Li, Zhang, and Liu 2016) constrains weight to +1, 0 and -1 and optimizes a threshold-based ternary function.”
A Lite Distributed Semantic Communication System for Internet of Things (2020, methodology): “A mix precision quantization by Li et al. [26] quantizes weights while keeping the activations at full-resolution.”
A review of convolutional neural network architectures and their optimizations (2022, background): “put forward holistic global pruning (Li et al. 2016).”
Relaxed Quantization for Discretized Neural Networks (2018, unknown): “…works that employ ﬁxed-point quantization: SR+DR (Gupta et al., 2015; Gysel et al., 2018), LR Net (Shayer et al., 2018), Jacob et al. (2017), TWN (Li et al., 2016), INQ (Zhou et al., 2017), BWN (Rastegari et al., 2016), XNOR-net (Rastegari et al., 2016), DoReFa (Zhou et al., 2016), HWGQ (Cai et…”
Extreme Compression for Pre-trained Transformers Made Simple and Efficient (2022, methodology): “We acknowledge that, in order to make a fair comparison with [61, 3], our investigations are based on the classical 1-bit [39] and 2-bit [27] algorithms and without quantization on layernorm [2].”
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review (2023, methodology): “In ternary weight nets (TWNs) [145] and trained ternary quantization (TTQ) [146], ternary weights instead of binary weights are used with 32-bit floatingpoint representation for activations that exhibit as low as 0.”
Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better (2021, background): “Additionally, binary quantization (and other quantization schemes like ternary [83], bit-shift based networks [110], etc.”
Self-Distillation: Towards Efficient and Compact Neural Networks (2021, methodology): “Li et al. propose the ternary weights neural networks, which constrains the weights to 0, +1 and -1 [61] Zhou et al propose the INQ method, targeting to convert convolutional neural networks into a low-precision model whose weights are constrained to be the powers of two [62].”
Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training With Non-IID Private Data (2020, methodology): “Another stream of research proposed themodel compression to reduce the communication cost required for the model parameter exchanges, which can be performed via several strategies, such as low-rank representation [17], model parameter quantization [17], [18], [19], neural network pruning [20], updat”
Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networks (2020, unknown): “The SPEQ training with the KD method is also compared with other KD training meth-ods for QDNNs in Table 7.”