Training DNNs with Hybrid Block Floating Point

By Mario Paulo Drumond, Tao Lin, Martin Jaggi, Babak Falsafi. 2018

121 citations recorded by the source corpus.

The wide adoption of DNNs has given birth to unrelenting computing requirements, forcing datacenter operators to adopt domain-specific accelerators to train them. These accelerators typically employ densely packed full-precision floating-point arithmetic to maximize performance per area. Ongoing research efforts seek to further increase that performance density by replacing floating-point with fixed-point arithmetic. However, a significant roadblock for these attempts has been fixed point's narrow dynamic range, which is insufficient for DNN training convergence. We identify block floating point (BFP) as a promising alternative representation since it exhibits wide dynamic range and enables the majority of DNN operations to be performed with fixed-point logic. Unfortunately, BFP alone introduces several limitations that preclude its direct applicability. In this work, we introduce HBFP, a hybrid BFP-FP approach, which performs all dot products in BFP and other operations in floating point. HBFP delivers the best of both worlds: the high accuracy of floating point at the superior hardware density of fixed point. For a wide variety of models, we show that HBFP matches floating point's accuracy while enabling hardware implementations that deliver up to 8.5 × higher throughput.

Subfields: Computer Networks and Communications, Electrical and Electronic Engineering, Hardware and Architecture, Efficiency & Compression, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Electrical and Electronic Engineering

In Hardware and Architecture

In Efficiency & Compression

In ML Systems & Hardware

Citations in the corpus

Stable and low-precision training for large-scale vision-language models (2023, background): “Another line of work is centered around 8-bit float data types which can be used to accelerate training of neural networks [21, 56, 62, 38, 3].”
8-bit Optimizers via Block-wise Quantization (2021, methodology): “The work most similar to our block-wise quantization is work on Hybrid Block Floating Point (HBFP) (Drumond et al., 2018) which uses a 24-bit fraction data type with a separate exponent for each tile in matrix multiplication to perform 24-bit matrix multiplication.”
Communication Compression for Decentralized Training (2018, methodology): “In Drumond et al. [2018], a compressed DNN training algorithm is proposed.”
Scaling Distributed Machine Learning with In-Network Aggregation (2019, unknown): “32-bit integers using a block floating-point-like approach [25], which is done efficiently at end hosts without impacting training accuracy.”
A Statistical Framework for Low-bitwidth Training of Deep Neural Networks (2020, background): “%) validation accuracy for training ResNet-50 [29] on ImageNet [26, 22].”