ZeroQ: A Novel Zero Shot Quantization Framework

By Yaohui Cai, Zi-Mei Yao, Zhen Dong, A. Gholami, Michael W. Mahoney, Kurt Keutzer. 2020

496 citations recorded by the source corpus.

Quantization is a promising approach for reducing the inference time and memory footprint of neural networks. However, most existing quantization methods require access to the original training dataset for retraining during quantization. This is often not possible for applications with sensitive or proprietary data, e.g., due to privacy and security concerns. Existing zero-shot quantization methods use different heuristics to address this, but they result in poor performance, especially when quantizing to ultra-low precision. Here, we propose \OURS, a novel zero-shot quantization framework to address this. \OURS enables mixed-precision quantization without any access to the training or validation data. This is achieved by optimizing for a Distilled Dataset, which is engineered to match the statistics of batch normalization across different layers of the network. \OURS supports both uniform and mixed-precision quantization. For the latter, we introduce a novel Pareto frontier based method to automatically determine the mixed-precision bit setting for all layers, with no manual search involved. We extensively test our proposed method on a diverse set of models, including ResNet18/50/152, MobileNetV2, ShuffleNet, SqueezeNext, and InceptionV3 on ImageNet, as well as RetinaNet-ResNet50 on the Microsoft COCO dataset. In particular, we show that \OURS can achieve 1.71\% higher accuracy on MobileNetV2, as compared to the recently proposed DFQ~\cite{nagel2019data} method. Importantly, \OURS has a very low computational overhead, and it can finish the entire quantization process in less than 30s (0.5\% of one epoch training time of ResNet50 on ImageNet). We have open-sourced the \OURS framework\footnote{https://github.com/amirgholami/ZeroQ}.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers (2022, background): “Recently, zero-shot quantization [9, 46] and post-training quantization (PTQ) [45, 38] are proposed to address the training-data access and compute requirement challenges since PTQ generally requires no (or minimal) retraining.”
Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion (2019, methodology): “While we have studied three datafree tasks in this work, the proposed paradigm of data-free knowledge distillation via model inversion easily scales to other applications, such as the task of data-free network quantization as independently studied in [19, 7].”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “2) Post-Training Quantization: An alternative to the expensive QAT method is Post-Training Quantization (PTQ) which performs the quantization and the adjustments of the weights, without any ﬁne-tuning [11, 24, 40, 60, 61, 68, 69, 88, 107, 140, 146, 170, 178, 273].”
The Knowledge Within: Methods for Data-Free Model Compression (2019, methodology): “ther insight into the success of the method and to the importance of the internal divergence measure. Additionally, we present data-free KD results for low precision models, which are not explored by [5], and show highly compressed models can recover from extreme accuracy degradation which enables m”
Up or Down? Adaptive Rounding for Post-Training Quantization (2020, background): “On the other hand, much attention has recently been dedicated to post-training quantization methods (Nagel et al., 2019; Cai et al., 2020; Choukroun et al., 2019; Banner et al., 2019), which can be more easily applied in practice.”
SqueezeLLM: Dense-and-Sparse Quantization (2023, methodology): “Quantization-Aware Training (QAT) requires retraining the model to adapt its weights to help recover accuracy after quantization [1,27,39, 51,53,54], whereas Post-Training Quantization (PTQ) does not involve retraining [4,31,35,40,55].”
A Statistical Framework for Low-bitwidth Training of Deep Neural Networks (2020, background): “Among inference quantization approaches, post training quantization usually does not require access to the partial/full training dataset, and it does not need to re-train/ﬁne-tune the quantized model [1, 2, 3, 4, 5].”
Fully Dynamic Inference With Deep Neural Networks (2020, background): “An alternative approach focuses on reducing computational overhead through weight quantization, where bit precision can be reduced without affecting accuracy [11], [12].”
HAWQV3: Dyadic Neural Network Quantization (2020, background): “As such, many prior quantization methods keep BN parameters in FP32 precision (e.g., (Cai et al., 2020; Chin et al., 2020; Choi et al., 2018; Dong et al., 2020; Park et al., 2018b; Zhang et al., 2018), just to name a few).”
Self-Distillation as Instance-Specific Label Smoothing (2020, background): “Recently, data-free distillation, a novel scenario in which the original data for the teacher is unavailable to students, has also been extensively studied [6, 7, 25, 40].”
CoDeNet: Efficient Deployment of Input-Adaptive Object Detection on Embedded FPGAs (2021, methodology): “Quantization [43][15][39][8][3] is a critical technique for efficiently deploying neural network models on embedded devices.”
LLM-QAT: Data-Free Quantization Aware Training for Large Language Models (2023, unknown): “Nevertheless, it is important to note that implementing this group-wise quantization method may necessitate specialized kernel design to fully realize its potential for actual speedup (Shen et al., 2020a; Cai et al., 2020).”
Memory-Free Generative Replay For Class-Incremental Learning (2021, methodology): “In this paper, we adopt the batch normalization loss [3] to reduce the domain gap and propose a diversity loss to address the homogenization problem.”
Post-Training Quantization for Vision Transformer (2021, methodology): “Most of the existing post-training quantization methods are designed for convolutional neural networks [3, 21, 30] or recurrent neural networks [36].”
SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers (2024, unknown): “ZeroQ (Cai et al., 2020) uses distillation to generate proxy input images for PTQ. SQuant (Guo et al., 2022) employs random samples with objectives based on Hessian spectrum sensitivity.”
HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks (2019, background): “Quantization [1, 6, 15, 36, 34, 17, 35, 8, 32, 33, 29, 9, 10, 5, 7] is a very promising approach to address this problem by reducing the memory bottleneck, thus allowing the use of lower precision computational units in hardware.”
BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction (2021, methodology): “Mixed precision also appears in PTQ, such as the Pareto frontier method in ZeroQ (Cai et al., 2020) and the Integer Programming method in AdaQuant (Hubara et al.”