Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science

By Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H. Nguyen, Madeleine Gibescu, Antonio Liotta. 2017

750 citations recorded by the source corpus.

Through the success of deep learning in various domains, artificial neural networks are currently among the most used artificial intelligence methods. Taking inspiration from the network properties of biological neural networks (e.g. sparsity, scale-freeness), we argue that (contrary to general practice) artificial neural networks, too, should not have fully-connected layers. Here we propose sparse evolutionary training of artificial neural networks, an algorithm which evolves an initial sparse topology (Erdős–Rényi random graph) of two consecutive layers of neurons into a scale-free topology, during learning. Our method replaces artificial neural networks fully-connected layers with sparse ones before training, reducing quadratically the number of parameters, with no decrease in accuracy. We demonstrate our claims on restricted Boltzmann machines, multi-layer perceptrons, and convolutional neural networks for unsupervised and supervised learning on 15 datasets. Our approach has the potential to enable artificial neural networks to scale up beyond what is currently possible. Artificial neural networks are artificial intelligence computing methods which are inspired by biological neural networks. Here the authors propose a method to design neural networks as sparse scale-free networks, which leads to a reduction in computational time required for training and inference.

Subfields: Artificial Intelligence, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Optimization & Training

Citations in the corpus

The State of Sparsity in Deep Neural Networks (2019, background): “, 2018), and a number of techniques have been proposed that draw inspiration from biological phenomena, and derive from evolutionary algorithms and neuromorphic computing (Guo et al., 2016; Bellec et al., 2017; Mocanu et al., 2018).”
Initialization and Regularization of Factorized Neural Layers (2021, methodology): “Best of (Mostafa & Wang, 2019; Mocanu et al., 2018; Bellec et al., 2018), obtained from Wang et al. (2020).”
Dynamic Model Pruning with Feedback (2020, methodology): “In dynamic pruning methods, the sparsity mask is readjusted during training according to different criteria (Mostafa & Wang, 2019; Mocanu et al., 2018).”
NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networks (2022, background): “Training sparse neural networks has been extensively explored (Bellec et al., 2018; Evci et al., 2020; Mocanu et al., 2018; Liu et al., 2021b; 2020; 2021a).”
Supermasks in Superposition (2020, methodology): “The Edge-Popup algorithm from [39] is used to obtain supermasks for various sparsities with a layer-wise budget from [35].”
Rigging the Lottery: Making All Tickets Winners (2019, methodology): “In Figure 9 we repeat the hyper-parameter sweep done for RigL in Figure 5-right, using SET and SNFS. Cosine schedule with ∆ T = 50 and α = 0 .”
Sparse Networks from Scratch: Faster Training without Losing Performance (2019, methodology): “SET also does not include the cross-layer redistribution of weights which we ﬁnd to be critical for good performance, as shown in our ablation study.”
Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization (2019, methodology): “Because both SET and ours reallocate parameters only intermittently (every few hundred training iterations), the computational overhead was negligible for the experiments pre- sented here4.”
TabNet: Attentive Interpretable Tabular Learning (2019, result): “We also compare to the state-of-the-art evolutionary sparsification algorithm (Mocanu et al. 2018) that integrates non-structured sparsity into training.”
Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training (2021, methodology): “We find the concept of In-Time Over-Parameterization useful (1) in exploring the expressibility of sparse training, especially for extreme sparsities, (2) in reducing training and inference costs (3) in understanding the underlying mechanism of dynamic sparse training (DST) (Mocanu et al., 2018; Evc”
Loss of plasticity in deep continual learning (2024, positive): “The idea of selective reinitialization is similar to the emerging idea of dynamic sparse training 110–112 .”
Stabilizing the Lottery Ticket Hypothesis (2019, background): “…(Lee et al., 2019), pruning throughout training (Zhu and Gupta, 2017; Gale et al., 2019; Lym et al., 2019; Molchanov et al., 2017; Louizos et al., 2018; Narang et al., 2017), and maintaining a sparse network with dynamic structure (Bellec et al., 2018; Mostafa and Wang, 2018; Mocanu et al., 2018).”
Low-Memory Neural Network Training: A Technical Report (2019, methodology): “The sparse training technique we leverage in this paper is dynamic sparse reparameterization (DSR), introduced by Mostafa and Wang [2019], which is a modification of SET [Mocanu et al., 2018].”
Machine Learning in Aerodynamic Shape Optimization (2022, background): “They are powerful and scalable, making them ideal for tackling large-scale and highly complex ML tasks [245, 246] (Fig.”
Pruning neural networks without any data by iteratively conserving synaptic flow (2020, background): “While there are a variety of approaches to compressing neural networks, such as novel design of micro-architectures [15, 16, 17], dimensionality reduction of network parameters [18, 19], and training of dynamic sparse networks [20, 21], in this work we will focus on neural network pruning.”
ResMLP: Feedforward Networks for Image Classification With Data-Efficient Training (2021, background): “Several works [37, 40, 59] have questioned if FCNs are also competitive on natural image datasets, such as CIFAR-10 [34].”
Soft Threshold Weight Reparameterization for Learnable Sparsity (2020, neutral): “, 1990; Hassibi & Stork, 1993), and magnitude-based pruning (Han et al., 2015; Guo et al., 2016; Zhu & Gupta, 2017; Frankle & Carbin, 2019; Gale et al., 2019; Mostafa & Wang, 2019; Wortsman et al., 2019; Bellec et al., 2018; Mocanu et al., 2018; Narang et al., 2019; Kusupati et al., 2018).”