Spatially-sparse convolutional neural networks

By Benjamin Graham. 2014

243 citations recorded by the source corpus.

Convolutional neural networks (CNNs) perform well on problems such as handwriting recognition and image classification. However, the performance of the networks is often limited by budget and time constraints, particularly when trying to train deep networks. Motivated by the problem of online handwriting recognition, we developed a CNN for processing spatially-sparse inputs; a character drawn with a one-pixel wide pen on a high resolution grid looks like a sparse matrix. Taking advantage of the sparsity allowed us more efficiently to train and test large, deep CNNs. On the CASIA-OLHWDB1.1 dataset containing 3755 character classes we get a test error of 3.82%. Although pictures are not sparse, they can be thought of as sparse by adding padding. Applying a deep convolutional network using sparsity has resulted in a substantial reduction in test error on the CIFAR small picture datasets: 6.28% on CIFAR-10 and 24.30% for CIFAR-100.

Subfields: Computer Vision and Pattern Recognition, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Model Architectures

In Efficiency & Compression

Citations in the corpus

Training Very Deep Networks (2015, neutral): “[5] and recently extended by Graham [32].”
Grid Long Short-Term Memory (2015, background): “-way interaction. We have seen the advantages of the cells compared to regular connections in solving tasks such as parity, addition and memorization. We 12 Test Error (%) Wan et al. [38] 0.28 Graham [11] 0.31 Untied 3-LSTM 0.32 Ciresan et al. [5] 0.35 Untied 3-LSTM with ReLU (*) 0.36 Mairar et al. ”
Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning (2016, background): “It is shown that this loss function makes use of unlabeled data and pushes the decision boundary to a less dense area of decision space.”
Fractional Max-Pooling (2014, methodology): “We used the representation for online characters described in (Graham, 2014); the characters were drawn with size 64 × 64 and additional features measuring the direction of the pen are added to produce an array of size 64 × 64 × 9 .”
BinaryConnect: Training Deep Neural Networks with binary weights during propagations (2015, methodology): “We do not use any data-augmentation (which can really be a game changer for this dataset [35]).”
All you need is a good init (2015, methodology): “For the FitNet-1 architecture, we have not experienced any difﬁculties training the network with any of the activation functions (ReLU, maxout, tanh), optimizers (SGD, RMSProp) or initialization (Xavier, MSRA, Ortho, LSUV), unlike the uniform initialization used in Romero et al. (2015).”
OctNet: Learning Deep 3D Representations at High Resolutions (2016, methodology): “A similar approach is presented in [15, 16] where sparse convolutions are reduced to matrix operations.”
Vote3Deep: Fast object detection in 3D point clouds using efficient convolutional neural networks (2016, background): “An alternative approach that takes advantage of sparse representations can be found in [10] and [11], in which sparse convolutions are applied to comparatively small 2D and 3D crops respectively.”
Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs (2017, methodology): “Our approach is largely inspired by Graham’s sparse convolutional networks [15, 16], which enable efficient shape analysis by storing a sparse set of non-trivial features instead of dense feature maps.”
Binarized Neural Networks (2016, methodology): “We do not use any preprocessing or data-augmentation (which can really be a game changer for this dataset (Graham, 2014)).”
More is Less: A More Complicated Network with Less Inference Complexity (2017, methodology): “In [8], a CNN model was proposed to process spatially-sparse inputs, which can be exploited to increase the speed of the evaluation process.”
PerforatedCNNs: Acceleration through Elimination of Redundant Convolutions (2015, background): “The resulting matrix of size X ′Y ′ × T is the output tensor V , up to a reshape.”
Sparse 3D convolutional neural networks (2015, methodology): “The sparse CNN algorithm from [3] can be tweaked to work efficiently on general lattices.”
Beyond Temporal Pooling: Recurrence and Temporal Convolutions for Gesture Recognition in Video (2015, result): “Leaky ReLUs seemed to work better than conventional ReLUs and showed promising results in other work [17, 6, 3, 30].”
A Bayesian encourages dropout (2014, methodology): “This layer-wise setting of the dropout rate is sometimes used in practice, e.g., (Graham, 2014).”
ReNet: A Recurrent Neural Network Based Alternative to Convolutional Networks (2015, methodology): “Test Error Model 0.28% [Wan et al., 2013] 0.31% [Graham, 2014a] [Zeiler and Fergus, 2013] (c) SVHN Table 2: Generalization errors obtained by the proposed ReNet along with those re-ported by previous works on each of the three datasets. denotes a convolutional neural network.”
OccuSeg: Occupancy-Aware 3D Instance Segmentation (2020, methodology): “While volumetric approaches [26, 25, 21] employ Spatially-Sparse Convolutional Networks (SSCN) [12, 2] to omit computations on inactive voxels using the sparse convolution technique.”
Deep Learning for 3D Point Clouds: A Survey (2019, background): “Later, Yan et al. [120] used the sparse convolutional network [134] to improve the inference efﬁciency of [108].”
Universum Prescription: Regularization Using Unlabeled Data (2015, methodology): “Although (Graham 2014) has the best results, we believe that by applying out universum prescription methods to their model design could also improve the results further.”