Sparse Convolutional Neural Networks

By Baoyuan Liu, Min Wang, Hassan Foroosh, Marshall F. Tappen, Marianna Pensky. 2015

924 citations recorded by the source corpus.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Model Architectures

In Efficiency & Compression

Citations in the corpus

Recent advances in convolutional neural networks (2015, methodology): “ve. HashNets may be used together with pruning to give even better parameter savings. 4.5. Sparse Convolution Recently, several attempts have been made to sparsify the weights of convolutional layers [190, 191]. Liu et al . [190] consider sparse representations of the basis lters, and achieve 90% sp”
MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep Networks (2017, background): “, each individual parameter is equally valuable) [3, 19, 32].”
Deep Fried Convnets (2014, background): “In parallel or after the first (technical report) version of this work, several researchers have attempted to create sparse networks by applying pruning or sparsity regularizers [8, 4, 26, 14].”
Variational Dropout Sparsifies Deep Neural Networks (2017, background): “Most recent works on sparse neural networks use pruning (Han et al., 2015b), elastic net regularization (Lebedev & Lempitsky, 2015; Liu et al., 2015; Scardapane et al., 2016; Wen et al., 2016) or composite techniques (Han et al., 2015a; Guo et al., 2016; Ullrich et al., 2017).”
Learning Structured Sparsity in Deep Neural Networks (2016, methodology): “[6] utilized group Lasso to constrain the structure scale of LRA.”
Learning Efficient Convolutional Networks through Network Slimming (2017, background): “Sparsity can be imposed on different level of structures [2, 37, 35, 29, 25], which yields considerable model-size compression and inference speedup.”
Selfless Sequential Learning (2018, methodology): “Differently from most of the compression based approaches [26, 1, 27], we argue that achieving sparsity in the network parameters is less significant in a sequential learning scenario than sparsity in the neural activations.”
Channel Pruning for Accelerating Very Deep Neural Networks (2017, neutral): “Sparse connection eliminates connections between neurons [17, 33, 29, 15, 14].”
Learning the Number of Neurons in Deep Networks (2016, methodology): “On ImageNet, with our Dec 8 -768 GS model and the (cid:96) 1 weight set to 0.0001 as in [Liu et al., 2015], this method yields 1.34M zero-valued parameters, compared to 7.74M for our approach, i.e., a 82% relative reduction in the number of individual parameters for our approach.”
Soft Weight-Sharing for Neural Network Compression (2017, background): “, 2015) or combining low rank decomposition with sparsity (Liu et al., 2015).”
LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks (2018, methodology): “Network parameter reduction: Considerable efforts have been devoted to reducing the number of parameters in an existing network [13, 12, 17, 36, 7, 8, 24, 27, 38, 51, 49, 4, 35, 28, 45, 10, 46].”
NISP: Pruning Networks Using Neuron Importance Score Propagation (2017, methodology): “Sparsity regularization terms have been use to learn sparse CNN structure in [23, 35, 33].”
Data-Driven Sparse Structure Selection for Deep Neural Networks (2017, background): “In spare learning, prior works [24] pursued the sparsity of weights.”
Designing Energy-Efficient Convolutional Neural Networks Using Energy-Aware Pruning (2016, methodology): “The proposed energy-efficient pruning algorithm can be combined with other techniques to further reduce the energy consumption, such as bitwidth reduction of weights or feature maps [14, 23, 24], weight sharing and Huffman coding [6], student-teacher learning [25], filter decomposition [26, 27] and ”
Adaptive Neural Networks for Efficient Inference (2017, methodology): “…second approach, an adaptive network selection method, takes a set of pre-trained DNNs, each with a different cost/accuracy trade-off, and arranges them in a directed acyclic graph (Trapeznikov & Saligrama, 2013; Wang et al., 2015), with the the cheapest model ﬁrst and the most expensive one last.”
More is Less: A More Complicated Network with Less Inference Complexity (2017, background): “Generally, most of these sparsity-based methods [22, 25, 31] integrate the sparsity property as a regularizer into the learning of parameters, which usually harms the performance of network.”
Design of Efficient Convolutional Layers using Single Intra-channel Convolution, Topological Subdivisioning and Spatial "Bottleneck" Structure (2016, methodology): “ncy of convolutional layer, several attempts have been made to study its redundancy. In both [7] and [8], low rank decomposition is utilized to achieve approximately two times speedup; The authors of [9] accomplish much higher efﬁciency improvement by obtaining more than 90% sparsity in the convolut”
Resource-Efficient Neural Architect (2018, background): “These include sparsity regularization [Liu et al., 2015], connection pruning [Han et al., 2015], and reducing the precision of weights and activations [Hubara et al., 2016].”
Domain-Adaptive Deep Network Compression (2017, background): “This process can be conducted while training [2, 38, 14, 20], or by analyzing the inﬂuence of each parameter once the network has been trained [22].”
3D Semantic Segmentation with Submanifold Sparse Convolutional Networks (2017, background): “4Our “sparse convolutional networks” are networks designed to operate on spatially-sparse input data; they do not have sparse parameters [12, 13].”