Understanding Locally Competitive Networks

By Rupesh Kumar Srivastava, Jonathan Masci, Faustino J. Gomez, J. Urgen Schmidhuber. 2014

41 citations recorded by the source corpus.

Recently proposed neural network activation functions such as rectified linear, maxout, and local winner-take-all have allowed for faster and more effective training of deep neural architectures on large and complex datasets. The common trait among these functions is that they implement local competition between small groups of computational units within a layer, so that only part of the network is activated for any given input pattern. In this paper, we attempt to visualize and understand this self-modularization, and suggest a unified explanation for the beneficial properties of such networks. We also show how our insights can be directly useful for efficiently performing retrieval over large datasets using neural networks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Highway Networks (2015, background): “Learning to route information through neural networks has helped to scale up their application to challenging problems by improving credit assignment and making training easier (Srivastava et al., 2015).”
Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks (2015, methodology): “The resulting code layer activations are then binarized via thresholding the ReLU activation which has been shown to be an effective information preserving technique (Srivastava et al., 2014) and provides a convenient form of semantic-hashing, allowing for linear time de-duplication .”
Dynamic Channel Pruning: Feature Boosting and Suppression (2018, background): “tions are incompatible with SGD, and resort to reinforcement learning for training. In contrast, (6) allows end-to-end training, as wta is a piecewise dierentiable and continuous function like ReLU. Srivastava et al. (2015) suggested that in general, a network is easier and faster to train for compl”
Geometric deep learning (2016, background): “This has opened up many interesting research directions to allow gradients to ﬂow more easily as in Highway and Residual Networks [83, 34], better behaved activation functions as the Exponential Linear Unit [18], and understanding the training process itself from a “model of models” standpoint [84].”