Highly Efficient Forward and Backward Propagation of Convolutional Neural Networks for Pixelwise Classification

By Hongsheng Li, Rui Zhao, Xiaogang Wang. 2014

87 citations recorded by the source corpus.

We present highly efficient algorithms for performing forward and backward propagation of Convolutional Neural Network (CNN) for pixelwise classification on images. For pixelwise classification tasks, such as image segmentation and object detection, surrounding image patches are fed into CNN for predicting the classes of centered pixels via forward propagation and for updating CNN parameters via backward propagation. However, forward and backward propagation was originally designed for whole-image classification. Directly applying it to pixelwise classification in a patch-by-patch scanning manner is extremely inefficient, because surrounding patches of pixels have large overlaps, which lead to a lot of redundant computation. The proposed algorithms eliminate all the redundant computation in convolution and pooling on images by introducing novel d-regularly sparse kernels. It generates exactly the same results as those by patch-by-patch scanning. Convolution and pooling operations with such kernels are able to continuously access memory and can run efficiently on GPUs. A fraction of patches of interest can be chosen from each training image for backward propagation by applying a mask to the error map at the last CNN layer. Its computation complexity is constant with respect to the number of patches sampled from the image. Experiments have shown that our proposed algorithms speed up commonly used patch-by-patch scanning over 1500 times in both forward and backward propagation. The speedup increases with the sizes of images and patches.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Deep Fried Convnets (2014, background): “Many other works have addressed the computational efficiency of convolutional networks in more specialized settings [13, 24].”
Deep learning for cellular image analysis (2019, background): “…for the Deep learning architectures often use iterative rounds of the three operations in a – c to produce low-dimensional representations of images. d , Dilations allow convolutional and pooling kernels to increase their spatial extent while keeping the number of parameters fixed 48,155 .”
Deep Contrast Learning for Salient Object Detection (2016, methodology): “‘im2col’ is a function implemented in Caffe to convert multi-channel feature maps to vectorized patches for improving the efficiency of convolutions.”
Dense Prediction on Sequences with Time-Dilated Convolutions for Speech Recognition (2016, methodology): “This was called filter rarefaction in [10], introduced as “d-regularly sparse kernels” in [13], and dubbed spatial dilated convolutions in [14].”
MultiNet: Real-time Joint Semantic Reasoning for Autonomous Driving (2016, unknown): “Fully Convolutional Networks (FCNs) were proposed to model semantic segmentation using a deep learning pipeline that is trainable end-to-end.”
Longitudinal multiple sclerosis lesion segmentation: Resource and challenge (2017, unknown): “This sampling technique speeds up the training of the CNNs for segmentation by using the sparse convolution method (Li et al., 2014).”
CNN-based Segmentation of Medical Imaging Data (2017, result): “This is supported by reports of larger patch sizes yielding better results [7, 27].”