Hybrid 8-bit Floating Point (HFP8) Training and Inference for Deep Neural Networks

By Xiao Sun, Jungwook Choi, Chia-Yu Chen, Naigang Wang, Swagath Venkataramani, Vijayalakshmi Srinivasan, Xiaodong Cui, Wei Zhang, Kailash Gopalakrishnan. 2019

255 citations recorded by the source corpus.

Subfields: Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

Stable and low-precision training for large-scale vision-language models (2023, background): “Another line of work is centered around 8-bit float data types which can be used to accelerate training of neural networks [21, 56, 62, 38, 3].”
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022, neutral): “For example, a common variant of this data type has 5 bits for the exponent and 2 bits for the fraction (Wang et al., 2018; Sun et al., 2019; Cambier et al., 2020; Mellempudi et al., 2019) and uses either no scaling constants or zeropoint scaling.”
8-bit Optimizers via Block-wise Quantization (2021, background): “The most common combination is 5 bits for the exponent and 2 bits for the fraction (Wang et al., 2018b; Sun et al., 2019; Cambier et al., 2020; Mellempudi et al., 2019) with either no normalization or min-max normalization.”
Sparse is Enough in Scaling Transformers (2021, methodology): “Our approach is complementary to model quantization studies [35, 38, 28] that use fewer bits for the weights.”
FP8 Formats for Deep Learning (2022, unknown): “[20] also investigates FP8 inference of networks trained in higher precision and introduces returning of batch normalization statistics to improve result accuracy.”
A Statistical Framework for Low-bitwidth Training of Deep Neural Networks (2020, unknown): “Various 8-bit numerical formats have been proposed, including INT8 [20, 21, 22, 23], FP8 [24, 25], 𝐅 𝑙 (𝐇 𝑙 ,𝚯 𝑙 ) 𝐉 (𝑙) 𝐊 (𝑙) ⊗ ⊗ 𝐇 (𝑙−1) 𝚯 (𝑙) 𝐇 (𝑙) 𝑄 𝑏 (෡∇ 𝐇 (𝑙) ) ෡∇ 𝚯 (𝑙)”
A TinyML Platform for On-Device Continual Learning With Quantized Latent Replays (2021, background): “IBM has recently proposed a specialized 8-bit format for training called HFP8 [46], but its effectiveness is still under investigation.”
Ultra-Low Precision 4-bit Training of Deep Neural Networks (2020, methodology): “MobileNetV2 is more difﬁcult to train in 4-bit due to lesser redundancy in its parameters[15].”
Unit Scaling: Out-of-the-Box Low-Precision Training (2023, methodology): “By ‘training in FP8’ we mean that matmuls are performed in FP8 (inputs are cast down to FP8, with outputs in higher precision) with wider formats typically used elsewhere, following the lead of Sun et al. (2019); Noune et al. (2022) and Micikevicius et al. (2022).”