F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization

By Qing Jin, Jian Ren, Richard Zhuang, Sumant Hanumante, Zhengang Li, Zhiyu Chen, Yanzhi Wang, Kai-Min Yang, Sergey Tulyakov. 2022

59 citations recorded by the source corpus.

Neural network quantization is a promising compression technique to reduce memory footprint and save energy consumption, potentially leading to real-time inference. However, there is a performance gap between quantized and full-precision models. To reduce it, existing quantization approaches require high-precision INT32 or full-precision multiplication during inference for scaling or dequantization. This introduces a noticeable cost in terms of memory, speed, and required energy. To tackle these issues, we present F8Net, a novel quantization framework consisting of only fixed-point 8-bit multiplication. To derive our method, we first discuss the advantages of fixed-point multiplication with different formats of fixed-point numbers and study the statistical behavior of the associated fixed-point numbers. Second, based on the statistical and algorithmic analysis, we apply different fixed-point formats for weights and activations of different layers. We introduce a novel algorithm to automatically determine the right format for each layer during training. Third, we analyze a previous quantization algorithm -- parameterized clipping activation (PACT) -- and reformulate it using fixed-point arithmetic. Finally, we unify the recently proposed method for quantization fine-tuning and our fixed-point approach to show the potential of our method. We verify F8Net on ImageNet for MobileNet V1/V2 and ResNet18/50. Our approach achieves comparable and better performance, when compared not only to existing quantization techniques with INT32 multiplication or floating-point arithmetic, but also to the full-precision counterparts, achieving state-of-the-art performance.

Subfields: Computer Vision and Pattern Recognition, Neurology, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

The case for 4-bit precision: k-bit Inference Scaling Laws (2022, background): “It has been studied before how the standard deviation of the input relates to the optimal exponent bit configuration for the FP8 data type (Jin et al., 2022).”
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022, neutral): “Jin et al. (2022) provide an excellent analysis of when certain ﬁxed point exponent/fraction bit widths are optimal for inputs with a particular standard deviation.”
Rethinking Vision Transformers for MobileNet Size and Speed (2022, methodology): “As applicable to CNNs, architecture search, pruning, and quantization techniques are also thoroughly investigated [35, 36, 52, 37, 9, 45, 48].”