Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware

By Shiwei Liu, Decebal Constantin Mocanu, Amarsagar Reddy Ramapuram Matavalam, Yulong Pei, Mykola Pechenizkiy. 2019

95 citations recorded by the source corpus.

Artificial neural networks (ANNs) have emerged as hot topics in the research community. Despite the success of ANNs, it is challenging to train and deploy modern ANNs on commodity hardware due to the ever-increasing model size and the unprecedented growth in the data volumes. Particularly for microarray data, the very high dimensionality and the small number of samples make it difficult for machine learning techniques to handle. Furthermore, specialized hardware such as graphics processing unit (GPU) is expensive. Sparse neural networks are the leading approaches to address these challenges. However, off-the-shelf sparsity-inducing techniques either operate from a pretrained model or enforce the sparse structure via binary masks. The training efficiency of sparse neural networks cannot be obtained practically. In this paper, we introduce a technique allowing us to train truly sparse neural networks with fixed parameter count throughout training. Our experimental results demonstrate that our method can be applied directly to handle high-dimensional data, while achieving higher accuracy than the traditional two-phase approaches. Moreover, we have been able to create truly sparse multilayer perceptron models with over one million neurons and to train them on a typical laptop without GPU (https://github.com/dcmocanu/sparse-evolutionary-artificial-neural-networks/tree/master/SET-MLP-Sparse-Python-Data-Structures), this being way beyond what is possible with any state-of-the-art technique.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networks (2022, background): “Training sparse neural networks has been extensively explored (Bellec et al., 2018; Evci et al., 2020; Mocanu et al., 2018; Liu et al., 2021b; 2020; 2021a).”
Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training (2021, background): “Liu et al. (2020c) found that DST gradually optimizes the initial sparse topology towards a completely different one.”
Smart Anomaly Detection in Sensor Systems (2020, background): “…⋅ Eﬃcient fusion strategies [111] Acceleration Circuitry Limits ⋅ FPGA-based optimization [112],[113] ⋅ Algorithm complexity reduction [113],[114],[115] Energy Eﬃciency Power Limits ⋅ Eﬃcient routing protocols [116], [117],[118],[119] ⋅ Smart topologies [117],[118] ⋅ Lightweight operating systems…”
Low-Memory Neural Network Training: A Technical Report (2019, background): “Similarly, the activations can be sparsiﬁed, as in (Liu et al., 2019, Rhu et al., 2018).”