Rethinking floating point for deep learning

By Jeff Johnson. 2018

155 citations recorded by the source corpus.

Reducing hardware overhead of neural networks for faster or lower power inference and training is an active area of research. Uniform quantization using integer multiply-add has been thoroughly investigated, which requires learning many quantization parameters, fine-tuning training or other prerequisites. Little effort is made to improve floating point relative to this baseline; it remains energy inefficient, and word size reduction yields drastic loss in needed dynamic range. We improve floating point to be more energy efficient than equivalent bit width integer hardware on a 28 nm ASIC process while retaining accuracy in 8 bits with a novel hybrid log multiply/linear add, Kulisch accumulation and tapered encodings from Gustafson's posit format. With no network retraining, and drop-in replacement of all math and float32 parameters via round-to-nearest-even only, this open-sourced 8-bit log float is within 0.9% top-1 and 0.2% top-5 accuracy of the original float32 ResNet-50 CNN model on ImageNet. Unlike int8 quantization, it is still a general purpose floating point arithmetic, interpretable out-of-the-box. Our 8/38-bit log float multiply-add is synthesized and power profiled at 28 nm at 0.96x the power and 1.12x the area of 8/32-bit integer multiply-add. In 16 bits, our log float multiply-add is 0.59x the power and 0.68x the area of IEEE 754 float16 fused multiply-add, maintaining the same signficand precision and dynamic range, proving useful for training ASICs as well.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In ML Systems & Hardware

Citations in the corpus

The Efficiency Misnomer (2021, methodology): “FLOPs: A widely used metric as the proxy for the computational cost of a model is the number of ﬂoating-point multiplication-and-addition operations (Johnson, 2018; Kim et al., 2021; Arnab et al., 2021; Tay et al., 2021a;b; Narayanan et al., 2021; Liu et al., 2021).”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, critical): “While several interesting works exist in this area [10, 26, 121, 135, 169], the proposed methods often require a lot of hyperparameter tuning, or they only work for a few NN models on relatively easy learning tasks.”
Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networks (2020, background): “However, 8-bit precision training remains an open challenge (Johnson, 2018; Kalamkar et al., 2019).”
FP8 Formats for Deep Learning (2022, unknown): “Additional number representations, such as log-format [15, 11], posits, and log with posit exponent values [8] have been proposed in literature but have not been adopted in practice because the demonstrated benefits have not been sufficient to justify new math pipeline hardware designs.”
Towards Fully 8-bit Integer Inference for the Transformer Model (2020, methodology): “Beyond this, INT8 is 10 × more energy efﬁcient [ Johnson, 2018 ] and saves much less chip area than the commonly used 32-bit ﬂoating point (FP32) in hardware design [ Sze et al. , 2017 ] .”