Speeding Up Neural Networks for Large Scale Classification using WTA Hashing

By Amir-Hossein Bakhtiary, Àgata Lapedriza, David Masip. 2015

10 citations recorded by the source corpus.

In this paper we propose to use the Winner Takes All hashing technique to speed up forward propagation and backward propagation in fully connected layers in convolutional neural networks. The proposed technique reduces significantly the computational complexity, which in turn, allows us to train layers with a large number of kernels with out the associated time penalty. As a consequence we are able to train convolutional neural network on a very large number of output classes with only a small increase in the computational cost. To show the effectiveness of the technique we train a new output layer on a pretrained network using both the regular multiplicative approach and our proposed hashing methodology. Our results showed no drop in performance and demonstrate, with our implementation, a 7 fold speed up during the training.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Deep Fried Convnets (2014, methodology): “Since then, hashing methods have also been advanced to reduce the number of parameters [6, 3].”
ACDC: A Structured Efficient Linear Layer (2015, background): “Hashing techniques have been proposed to reduce the number of parameters (Chen et al., 2015; Bakhtiary et al., 2015).”