FBGEMM: Enabling High-Performance Low-Precision Deep Learning Inference

By Daya Shanker Khudia, Jianyu Huang, Protonu Basu, Summer Deng, Haixin Liu, Jongsoo Park, M. Smelyanskiy. 2021

69 citations recorded by the source corpus.

Deep learning models typically use single-precision (FP32) floating point data types for representing activations and weights, but a slew of recent research work has shown that computations with reduced-precision data types (FP16, 16-bit integers, 8-bit integers or even 4- or 2-bit integers) are enough to achieve same accuracy as FP32 and are much more efficient. Therefore, we designed fbgemm, a high-performance kernel library, from ground up to perform high-performance quantized inference on current generation CPUs. fbgemm achieves efficiency by fusing common quantization operations with a high-performance gemm implementation and by shape- and size-specific kernel code generation at runtime. The library has been deployed at Facebook, where it delivers greater than 2x performance gains with respect to our current production baseline.

Subfields: Computer Networks and Communications, Computer Vision and Pattern Recognition, Hardware and Architecture, Efficiency & Compression, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Computer Vision and Pattern Recognition

In Hardware and Architecture

In Efficiency & Compression

In ML Systems & Hardware

Citations in the corpus

Stable and low-precision training for large-scale vision-language models (2023, methodology): “However, we find the best trade-off of simplicity and performance is from using i) row-wise quantization [30] for the inputs and gradients and ii) tensor-wise quantization for the weights.”
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022, positive): “If one views matrix multiplication as 1x1 convolution, vector-wise quantization is equivalent to channel-wise quantization for convolution combined with row quantization (Khudia et al., 2021).”
SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (2023, unknown): “The FBGEMM package [KHB + 21] proposed a format in which certain “outlier” weights are quantized separately, to reduce their impact on normalization.”
Low-Precision Hardware Architectures Meet Recommendation Model Inference at Scale (2021, methodology): “This leads to our development of FBGEMM [5, 6, 36] library to support the fp16 and int8 operators on Intel CPUs, which has been the backend for the low-precision operators in both Caffe2 and PyTorch.”