Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks

By Urs Köster, T. Webb, Xin Wang, Marcel Nassar, Arjun K. Bansal, William Constable, Oguz H. Elibol, Stewart Hall, Luke Hornof, Amir Khosrowshahi and 3 more. 2017

278 citations recorded by the source corpus.

Deep neural networks are commonly developed and trained in 32-bit floating point format. Significant gains in performance and energy efficiency could be realized by training and inference in numerical formats optimized for deep learning. Despite advances in limited precision inference in recent years, training of neural networks in low bit-width remains a challenging problem. Here we present the Flexpoint data format, aiming at a complete replacement of 32-bit floating point format training and inference, designed to support modern deep network topologies without modifications. Flexpoint tensors have a shared exponent that is dynamically adjusted to minimize overflows and maximize available dynamic range. We validate Flexpoint by training AlexNet, a deep residual network and a generative adversarial network, using a simulator implemented with the neon deep learning framework. We demonstrate that 16-bit Flexpoint closely matches 32-bit floating point in training all three models, without any need for tuning of model hyperparameters. Our results suggest Flexpoint as a promising numerical format for future hardware for training and inference.

Subfields: Computational Mathematics, Hardware and Architecture, Statistical and Nonlinear Physics, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mathematics

In Hardware and Architecture

In Statistical and Nonlinear Physics

In Efficiency & Compression

Citations in the corpus

Zero-Shot Text-to-Image Generation (2021, methodology): “This can be seen as a practical alternative to a more general framework for mixed-precision training called Flexpoint (Köster et al., 2017), with the advantage that specialized GPU kernels are not required.”
QUOTIENT: Two-Party Secure Neural Network Training and Prediction (2019, background): “However, for the task of training, there are few works that perform all operations in fixed-point [20, 23, 33, 39, 57].”
Demystifying Parallel and Distributed Deep Learning (2018, methodology): “inent data representation for gradient (or parameter) compression is quantization, i.e., mapping continuous information into buckets that represent sets of values (usually ranges). It has been shown [Köster et al. 2017] that the distributions of parameter and gradient values are narrowly dispersed (”
Training Deep Neural Networks with 8-bit Floating Point Numbers (2018, background): “Flexpoint [13] and DFP [4] demonstrated a format with a 16-bit mantissa and a shared exponent to train large neural networks with full-precision accuracy.”
MLPerf Training Benchmark (2019, background): “With the introduction of systems with varying numerics (Abadi et al., 2016; Banner et al., 2018; Kster et al., 2017; Micikevicius et al., 2018) and performance optimization techniques, ML benchmarks cannot afford to omit accuracy measures.”
A Study of BFLOAT16 for Deep Learning Training (2019, background): “Literature shows that various different reduced precision data representations have been investigated which can be broadly classiﬁed into two types: the more standard ﬂoating-point based formats [28, 15, 12] and custom ﬁxed point based formats [37, 7, 18, 22, 23].”
Data-Free Quantization Through Weight Equalization and Bias Correction (2019, methodology): “Other approaches use low-bit floating point operations instead of integer operations, or other custom quantization implementations [8, 17, 24, 35].”
Pruning and Quantization for Deep Neural Network Acceleration: A Survey (2021, methodology): “However, it requires specialized hardware including 16-bit fixed-point [83], 16-bit flex point [122], and 12-bit operations using dynamic fixed-point format (DFXP) [48].”
Norm matters: efficient and accurate normalization schemes in deep networks (2018, background): “Optimized hardware was designed to leverage benefits of low-precision arithmetic and memory operations, with the promise of better, more efficient implementations [21].”
Scaling Distributed Machine Learning with In-Network Aggregation (2019, unknown): “Further, others have explored block ﬂoating-point representations [25,53], where a single exponent is shared by multiple tensor elements, reducing the amount of computation required to perform tensor operations.”
Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networks (2020, background): “While 32-bit ﬂoating point (FP32) is the most common data format for neural network training, recent hardware have leveraged techniques that allow for training with 16-bit data formats (K¨oster et al., 2017; Micikevicius et al., 2018).”
NVIDIA Tensor Core Programmability, Performance & Precision (2018, methodology): “Flexpoint uses fixed-point multiplications and a shared exponent to allow a large dynamic range.”
Training DNNs with Hybrid Block Floating Point (2018, methodology): “ZipML [15], DoReFa [6], and Flexpoint [5] train DNNs with end-to-end low precision.”
Mixed Precision Training of Convolutional Neural Networks using Integer Operations (2018, background): “Köster et al. (2017) have shown that a ﬁxed point numerical format designed for deep neural networks (Flexpoint), out-performs FP16 and achieves numerical parity with FP32 across a wide set of applications.”
A Configurable Cloud-Scale DNN Processor for Real-Time AI (2018, background): “Extensive literature (e.g., [11]–[14]) has shown that deep neural networks are highly compressible in data types and sparsity.”