Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning

By Elias Frantar, Dan Alistarh. 2022

407 citations recorded by the source corpus.

We consider the problem of model compression for deep neural networks (DNNs) in the challenging one-shot/post-training setting, in which we are given an accurate trained model, and must compress it without any retraining, based only on a small amount of calibration input data. This problem has become popular in view of the emerging software and hardware support for executing models compressed via pruning and/or quantization with speedup, and well-performing solutions have been proposed independently for both compression approaches. In this paper, we introduce a new compression framework which covers both weight pruning and quantization in a unified setting, is time- and space-efficient, and considerably improves upon the practical performance of existing post-training methods. At the technical level, our approach is based on an exact and efficient realization of the classical Optimal Brain Surgeon (OBS) framework of [LeCun, Denker, and Solla, 1990] extended to also cover weight quantization at the scale of modern DNNs. From the practical perspective, our experimental results show that it can improve significantly upon the compression-accuracy trade-offs of existing post-training methods, and that it can enable the accurate compound application of both pruning and quantization in a post-training setting.

Subfields: Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Efficiency & Compression

In Optimization & Training

Citations in the corpus

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (2022, methodology): “The GPTQ algorithm starts from a similar strategy to the recently-proposed Optimal Brain Quantization (OBQ) layerwise solver [8].”
SliceGPT: Compress Large Language Models by Deleting Rows and Columns (2024, methodology): “GPTQ (Frantar et al., 2022) has solved this issue by quantizing (representing the parameter using lower precision) the weight matrix of LLMs using a column-by-column scheme and updating all not-yet-quantized weights in the next columns.”
QuIP: 2-Bit Quantization of Large Language Models With Guarantees (2023, methodology): “[7] Elias Frantar, Sidak Pal Sing, and Dan Alistarh.”
SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (2023, positive): “Most PTQ methods, such as AdaRound [NAVB + 20], BitSplit [WCHC20], AdaQuant [HNH + 21], BRECQ [LGT + 21], or OBQ [FSA22] were designed for vision models or small-scale language models, with less than 100M parameters.”
SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot (2023, methodology): “The currently most accurate post-training method, OBC [11], exhibits runtime scaling to the 4th power of a transformer’s hidden dimension, while taking > 1 hour to compress a 100M parameter model.”
A Comprehensive Overview of Large Language Models (2023, neutral): “…for the alignment between vision and language modalities: in the first stage, vision-language representation learning is bootstrapped from a frozen visual encoder; and in the second stage, a frozen LLM bootstraps vision-to-language generative learning for zero-shot image-to-text generation.”
A Survey on Knowledge Distillation of Large Language Models (2024, methodology): “…(Frantar et al., 2023; Dettmers et al., 2022; Kim et al., 2023c; Tao et al., 2022b; Yao et al., 2022; Xiao et al., 2023), Parameter Pruning (Ma et al., 2023d; Zhang et al., 2023h; Frantar and Alistarh, 2023), and Low-Rank Approximation (Xu et al., 2023g; Li et al., 2023l) are commonly utilized.”
A Simple and Effective Pruning Approach for Large Language Models (2023, methodology): “To scale existing second-order based approaches [21] to LLMs, an efficient weight update procedure was proposed that iterates between weight removal and weight update at each layer.”
Pre-Trained Language Models for Text Generation: A Survey (2022, methodology): “To find optimal quantized weights that minimize a layerwise reconstruction loss, GPTQ [59] improves the original optimal brain quantization [58] method by fixing the quantization order of weights for all rows.”