Soft Threshold Weight Reparameterization for Learnable Sparsity

By Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham M. Kakade, Ali Farhadi. 2020

274 citations recorded by the source corpus.

Sparsity in Deep Neural Networks (DNNs) is studied extensively with the focus of maximizing prediction accuracy given an overall parameter budget. Existing methods rely on uniform or heuristic non-uniform sparsity budgets which have sub-optimal layer-wise parameter allocation resulting in a) lower prediction accuracy or b) higher inference cost (FLOPs). This work proposes Soft Threshold Reparameterization (STR), a novel use of the soft-threshold operator on DNN weights. STR smoothly induces sparsity while learning pruning thresholds thereby obtaining a non-uniform sparsity budget. Our method achieves state-of-the-art accuracy for unstructured sparsity in CNNs (ResNet50 and MobileNetV1 on ImageNet-1K), and, additionally, learns non-uniform budgets that empirically reduce the FLOPs by up to 50%. Notably, STR boosts the accuracy over existing results by up to 10% in the ultra sparse (99%) regime and can also be used to induce low-rank (structured sparsity) in RNNs. In short, STR is a simple mechanism which learns effective sparsity budgets that contrast with popular heuristics. Code, pretrained models and sparsity budgets are at this https URL.

Subfields: Artificial Intelligence, Civil and Structural Engineering, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Civil and Structural Engineering

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, background): “, 2018) and convolutional models for computer vision (Zhu & Gupta, 2018; Elsen et al., 2019; Evci et al., 2020; Kusupati et al., 2020).”
Rigging the Lottery: Making All Tickets Winners (2019, neutral): “Methods with superscript ‘*’ indicates reported results in corresponding papers (except DNW results, which is obtained from (Kusupati et al., 2020)).”
Are We Overfitting to Experimental Setups in Recognition (2020, background): “Previous works have primarily focused on efficient inference [35, 20, 24] and some on training costs [9].”
In the Wild: From ML Models to Pragmatic ML Systems (2020, background): “Pragmatic systems must measure this compute over the duration of their lifetime; not just measuring inference cost [34, 20, 24] but also update cost [10].”
Effective Sparsification of Neural Networks with Global Sparsity Constraint (2021, unknown): “DNW [35], GMP [41], STR [18] are state-of-the-art methods on dense-to-sparse training.”
A Simple and Effective Pruning Approach for Large Language Models (2023, methodology): “Existing pruning methods usually require either modifications to the training procedure (Sanh et al., 2020; Kusupati et al., 2020), retraining the pruned networks to regain accuracy (Liu et al., 2019; Zhou et al., 2023), or an even more computationally intensive iterative retraining process (Renda…”
Gradient Flow in Sparse Neural Networks and How Lottery Tickets Win (2020, methodology): “Pruning Pruning is used commonly in NN literature to obtain sparse networks (Mozer and Smolensky 1989; Han et al. 2015; Kusupati et al. 2020).”
Learning N: M Fine-grained Structured Sparse Neural Networks From Scratch (2021, methodology): “And STR (Kusupati et al., 2020) shows that training the model with non-uniform sparsity can improve the performance consistently, thus the SR-STE can extend the sparsity with non-uniform structural sparsity setting ( e . g ., mixed N : M ﬁne-grained structural sparsity).”
Layer-adaptive Sparsity for the Magnitude-based Pruning (2020, background): “We note that there is a line of results on the trainable layerwise sparsity ; we refer the interested readers to the recent work of Kusupati et al. (2020) for a concise survey.”
Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training (2021, methodology): “, 2018a) or a partial converged training time (partial dense over-parameterization) (Louizos et al., 2017; Zhu & Gupta, 2017; Gale et al., 2019; Savarese et al., 2019; Kusupati et al., 2020; You et al., 2019).”
How Well Do Sparse ImageNet Models Transfer? (2021, methodology): “For regularization methods, we consider the leading Soft Threshold Weight Reparametrization (STR) [41], and Alternating Compression/Decompression (AC/DC) [57] methods.”
M-FAC: Efficient Matrix-Free Approximations of Second-Order Information (2021, unknown): “Table 1 presents gradual pruning results for MobileNetV1STR/ImageNet at 89% sparsity and ResNet50/ImageNet at 95% sparsity, relative to other state-of-theart methods.”