Deep Rewiring: Training very sparse deep networks

By Guillaume Bellec, David Kappel, Wolfgang Maass, Robert A. Legenstein. 2017

311 citations recorded by the source corpus.

Neuromorphic hardware tends to pose limits on the connectivity of deep networks that one can run on them. But also generic hardware and software implementations of deep learning run more efficiently for sparse networks. Several methods exist for pruning connections of a neural network after it was trained without connectivity constraints. We present an algorithm, DEEP R, that enables us to train directly a sparsely connected neural network. DEEP R automatically rewires the network during supervised training so that connections are there where they are most needed for the task, while its total number is all the time strictly bounded. We demonstrate that DEEP R can be used to train very sparse feedforward and recurrent neural networks on standard benchmark tasks with just a minor loss in performance. DEEP R is based on a rigorous theoretical foundation that views rewiring as stochastic sampling of network configurations from a posterior.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Control and Systems Engineering, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Efficiency & Compression

Citations in the corpus

The State of Sparsity in Deep Neural Networks (2019, background): “…prune weights and convolutional ﬁlters (Lin et al., 2017; He et al., 2018), and a number of techniques have been proposed that draw inspiration from biological phenomena, and derive from evolutionary algorithms and neuromorphic computing (Guo et al., 2016; Bellec et al., 2017; Mocanu et al., 2018).”
Initialization and Regularization of Factorized Neural Layers (2021, methodology): “Best of (Mostafa & Wang, 2019; Mocanu et al., 2018; Bellec et al., 2018), obtained from Wang et al. (2020).”
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (2018, background): “Bellec et al. (2018) train with sparse networks and replace weights that reach zero with new random connections.”
Dynamic Model Pruning with Feedback (2020, methodology): “We omit the comparison to other dynamic reparameterization methods, as DSR can outperform DeepR (Bellec et al., 2018) and SET (Mocanu et al., 2018) by a noticeable margin (Mostafa & Wang, 2019). overhead.”
NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networks (2022, background): “Training sparse neural networks has been extensively explored (Bellec et al., 2018; Evci et al., 2020; Mocanu et al., 2018; Liu et al., 2021b; 2020; 2021a).”
Rigging the Lottery: Making All Tickets Winners (2019, methodology): “Following this, Deep Rewiring (DeepR) (Bellec et al., 2018) augmented Stochastic Gradient Descent (SGD) with a random walk in parameter space.”
Sparse Networks from Scratch: Faster Training without Losing Performance (2019, methodology): “Bellec et al. (2018) introduce DEEP-R, which takes a Bayesian perspective and performs sampling for prune and regrowth decisions – sampling sparse network conﬁgurations from a posterior.”
Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization (2019, methodology): “To the best of our knowledge, only three methods, namely DeepR (Bellec et al., 2017), SET (Mocanu et al., 2018) and ours, strictly impose, throughout the entire course of training, a fixed small parameter budget, one that is equal to the size of the final sparse model for inference.”
Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training (2021, background): “Bellec et al. (2018) proposed Deep Rewiring to train sparse neural networks with a strict connectivity constraint by sampling sparse configurations and weights from a posterior distribution.”
Picking Winning Tickets Before Training by Preserving Gradient Flow (2020, background): “Representative works, such as Bellec et al. (2018); Mocanu et al.”
The Lottery Ticket Hypothesis: Training Pruned Neural Networks (2018, background): “ohen et al. [7] observe that convolutional ﬁlters are sensitive to initialization (“The Filter Lottery”); after training, they randomly reinitialize unimportant ﬁlters. During training. Bellac et al. [4] train with sparse networks and replace weights that reach zero with new random connections. Srin”
Learning N: M Fine-grained Structured Sparse Neural Networks From Scratch (2021, methodology): “Deep-Rewiring (Bellec et al., 2017), on the other hand, is a typical one-stage method, which takes a Bayesian perspective and samples sparse network connections from a posterior, however is computationally expensive and challenging to be applied to large-scale tasks.”
Pruning neural networks without any data by iteratively conserving synaptic flow (2020, background): “While there are a variety of approaches to compressing neural networks, such as novel design of micro-architectures [15, 16, 17], dimensionality reduction of network parameters [18, 19], and training of dynamic sparse networks [20, 21, 22], in this work we will focus on neural network pruning.”
Low-Memory Neural Network Training: A Technical Report (2019, background): “The imposition of sparsity throughout training has been investigated by Bellec et al. (2018), Golub et al. (2019), Liu et al. (2021), Mocanu et al. (2018), Mostafa and Wang (2019).”
Soft Threshold Weight Reparameterization for Learnable Sparsity (2020, unknown): “This is referred to as sparse-to-sparse training and a lot of recent work (Mostafa & Wang, 2019; Bellec et al., 2018; Evci et al., 2019; Lee et al., 2019; Dettmers & Zettlemoyer, 2019) aims to do sparse-to-sparse training using techniques which include re-allocation of weights to improve accuracy.”
Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers (2020, methodology): “To avoid the expensive pruning and fine-tuning iterations, many sparse training methods (Mocanu et al., 2018; Bellec et al., 2017; Mostafa & Wang, 2019; Dettmers & Zettlemoyer, 2019) have been proposed, where the network pruning is conducted during the training process.”
Stabilizing the Lottery Ticket Hypothesis (2019, background): “…(Lee et al., 2019), pruning throughout training (Zhu and Gupta, 2017; Gale et al., 2019; Lym et al., 2019; Molchanov et al., 2017; Louizos et al., 2018; Narang et al., 2017), and maintaining a sparse network with dynamic structure (Bellec et al., 2018; Mostafa and Wang, 2018; Mocanu et al., 2018).”
Long short-term memory and Learning-to-learn in networks of spiking neurons (2018, methodology): “DEEP R converges theoretically to an optimal network configuration by continuously updating the set of active connections [23, 3, 4].”
Effective Sparsification of Neural Networks with Global Sparsity Constraint (2021, methodology): “, magnitude of the weight [9, 10, 8, 41, 6, 28, 1, 26, 35], scores based on Hessian, momentum or gradient [5, 20, 39, 19, 11, 4]) to evaluate the importance of the weight, kernel or filter and then remove the unimportant ones.”