Overcoming Oscillations in Quantization-Aware Training

By Markus Nagel, Marios Fournarakis, Yelysei Bondarenko, Tijmen Blankevoort. 2022

171 citations recorded by the source corpus.

When training neural networks with simulated quantization, we observe that quantized weights can, rather unexpectedly, oscillate between two grid-points. The importance of this effect and its impact on quantization-aware training (QAT) are not well-understood or investigated in literature. In this paper, we delve deeper into the phenomenon of weight oscillations and show that it can lead to a significant accuracy degradation due to wrongly estimated batch-normalization statistics during inference and increased noise during training. These effects are particularly pronounced in low-bit ($\leq$ 4-bits) quantization of efficient networks with depth-wise separable layers, such as MobileNets and EfficientNets. In our analysis we investigate several previously proposed QAT algorithms and show that most of these are unable to overcome oscillations. Finally, we propose two novel QAT algorithms to overcome oscillations during training: oscillation dampening and iterative weight freezing. We demonstrate that our algorithms achieve state-of-the-art accuracy for low-bit (3&4 bits) weight and activation quantization of efficient architectures, such as MobileNetV2, MobileNetV3, and EfficentNet-lite on ImageNet. Our source code is available at {https://github.com/qualcomm-ai-research/oscillations-qat}.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Sharpness-aware Quantization for Deep Neural Networks (2021, methodology): “Compared with recent studies [10, 54] that mitigates oscillations incurred by implicit stochasticity of the straight-through estimator (STE) [2] to stabilize optimization, our SAQ instead reduces the effect of adversarial perturbations and quantization noises by directly minimizing the perturbed qua”
QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks (2024, background): “QAT focuses on training models that are more “quantizable” but usually requires training models from scratch (Nagel et al., 2022; Xi et al., 2023).”
OLLA: Optimizing the Lifetime and Location of Arrays to Reduce the Memory Usage of Neural Networks (2022, background): “However, these all come at the cost of decreasing the accuracy of the model, or increasing the time it takes to train it, or both [49, 7, 57].”