DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures

By Huanrui Yang, Wujie Wen, Hai Helen Li. 2019

115 citations recorded by the source corpus.

In seeking for sparse and efficient neural network models, many previous works investigated on enforcing L1 or L0 regularizers to encourage weight sparsity during training. The L0 regularizer measures the parameter sparsity directly and is invariant to the scaling of parameter values, but it cannot provide useful gradients, and therefore requires complex optimization techniques. The L1 regularizer is almost everywhere differentiable and can be easily optimized with gradient descent. Yet it is not scale-invariant, causing the same shrinking rate to all parameters, which is inefficient in increasing sparsity. Inspired by the Hoyer measure (the ratio between L1 and L2 norms) used in traditional compressed sensing problems, we present DeepHoyer, a set of sparsity-inducing regularizers that are both differentiable almost everywhere and scale-invariant. Our experiments show that enforcing DeepHoyer regularizers can produce even sparser neural network models than previous works, under the same accuracy level. We also show that DeepHoyer can be applied to both element-wise and structural pruning.

Subfields: Artificial Intelligence, Computational Mechanics, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

Dan Alistarh

In Efficiency & Compression

Citations in the corpus

AutoGrow: Automatic Layer Growing in Deep Convolutional Networks (2019, neutral): “Structure pruning and growing were proposed for different goals, such as efficient inference [7, 8, 11, 13, 14, 18, 19, 24, 25, 27, 38–40], lifelong learning [41] and model adaptation [10, 29].”
Inducing and Exploiting Activation Sparsity for Fast Neural Network Inference (2020, methodology): “It is popular for compressed sensing, and has only recently been applied for weight sparsiﬁcation (Yang et al., 2019); to our knowledge, we are the ﬁrst to investigate it for activation sparsity.”
How Well Do Sparse ImageNet Models Transfer? (2021, background): “These mechanisms are very diverse, from surrogates of (cid:96) 0 and (cid:96) 1 - regularization [41,67], to variational methods [53], to meth-ods inspired by compressive sensing mechanisms such as Iterative Hard Thresholding (IHT) [32,33,45,57].”