WoodFisher: Efficient Second-Order Approximation for Neural Network Compression

By Sidak Pal Singh, Dan Alistarh. 2020

209 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (2022, background): “Finally, Optimal Brain Quantization (OBQ) [8] generalizes the classic Optimal Brain Surgeon (OBS) second-order weight pruning framework [10, 29, 7] to apply to quantization.”
Training Neural Networks with Fixed Sparse Masks (2021, background): “…learning, e.g. as a measure of parameter importance [21], as a preconditioner in gradient descent [4, 37, 34], as a way to measure the amount of “information” in each parameter of a neural network [1], or as a way to decide which parameters to prune when performing model compression [43, 10, 47].”
SliceGPT: Compress Large Language Models by Deleting Rows and Columns (2024, methodology): “…limitation posed by OBS, recent research has explored two approaches: approximating the inverse of the Hessian matrix such as WoodFisher (Singh & Alistarh, 2020) or applying it separately to each layer such as in Optimal Brain Compression (OBC, Frantar & Alistarh, 2022), known as…”
Group Fisher Pruning for Practical Network Compression (2021, background): “WoodFisher (Singh & Alistarh, 2020) approximates the inverse of Hessian matrix by the Woodbury matrix identity and improves unstructured pruning based on OBD/OBS.”
A Simple and Effective Pruning Approach for Large Language Models (2023, background): “Existing popular solvers [21, 60] for the layer-wise reconstruction problem rely on heavy computation of second-order Hessian inverses, which do not scale to the large hidden state size of LLMs.”
M-FAC: Efficient Matrix-Free Approximations of Second-Order Information (2021, methodology): “Singh and Alistarh [37] showed that it can be scaled to DNNs by blockwise approximation, and that it provides better approximations of the loss than K-FAC-based [44], or diagonal [39] approximations [7], leading to more accurate pruning.”
How Well Do Sparse ImageNet Models Transfer? (2021, methodology): “One of the main beneﬁts of sparse models is that they can provide inference speed-ups when executed on sparsity-aware runtimes [11, 40, 57, 63].”
Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning (2022, unknown): “The optimal solution is then derived by ﬁrst ﬁnding the optimal solution to δ p via setting the derivative ∂L/∂ δ p to zero and then substituting this solution back into L and solving for λ ; please see e.g. [13, 39] for examples.”
SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot (2023, methodology): “This is in stark contrast to smaller vision models which can usually be pruned via simple magnitude to 50% or more at very little loss of accuracy [43, 11].”