Accurate and Efficient 2-bit Quantized Neural Networks

By Jungwook Choi, Swagath Venkataramani, Vijayalakshmi Srinivasan, Kailash Gopalakrishnan, Zhuo Wang, Pierce I-Jen Chuang. 2019

172 citations recorded by the source corpus.

Subfields: Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022, neutral): “…different bit-widths have been studied: 1-bit methods (Courbariaux and Bengio, 2016; Rastegari et al., 2016; Courbariaux et al., 2015), 2 to 3-bit (Zhu et al., 2017; Choi et al., 2019), 4-bits (Li et al., 2019), more bits (Courbariaux et al., 2014), or a variable amount of bits (Gong et al., 2019).”
8-bit Optimizers via Block-wise Quantization (2021, methodology): “There are methods that use 1-bit (Courbariaux and Bengio, 2016; Rastegari et al., 2016; Courbariaux et al., 2015), 2-bit/3 values (Zhu et al., 2017; Choi et al., 2019), 4-bits (Li et al., 2019), more bits (Courbariaux et al., 2014), or a variable amount of bits (Gong et al., 2019).”
Ultra-Low Precision 4-bit Training of Deep Neural Networks (2020, background): “While the best training results are achieved using FP8, a number of new ideas have enabled inference precision to be scaled aggressively to 4-bit integer representations using quantization-aware training techniques [20–22 , 25].”
Low-Memory Neural Network Training: A Technical Report (2019, methodology): “Another line of work aims to train very low-precision – even binary (1-bit) – neural networks from scratch, such as in (Choi et al., 2019, Courbariaux et al., 2016, Hubara et al., 2018, Zhu et al., 2017).”
Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation (2021, background): “Quantization can be further classified to uniform quantization [7, 8, 14, 19, 20, 50] and nonuniform quantization [26, 34, 46, 47].”
Hybrid 8-bit Floating Point (HFP8) Training and Inference for Deep Neural Networks (2019, background): “In addition, for low-precision inference, ﬁxed point techniques involving costly retraining of networks for ultra-short bit-widths [11, 12, 13] as well as post-training quantization for simpler deployment of the INT8/INT4 inference models [7, 14, 15, 16] have been extensively explored, but the…”
Distribution Adaptive INT8 Quantization for Training CNNs (2021, unknown): “Many literatures (Jacob et al. 2018; Wang et al. 2018b; Hubara et al. 2016; Choi et al. 2019; Jain et al. 2019) have demonstrated that low bit-width quantization can be utilized to speed up the inference stage (weight or activation) with minor hurt of accuracy.”