On the piecewise analysis of networks of linear threshold neurons

By Richard Hans Robert Hahnloser. 1998

The computational abilities of recurrent networks of neurons with a linear activation function above threshold are analyzed. These networks selectively realise a linear mapping of their input. Using this property, the dynamics as well as the number and the stability of stationary states can be investigated. The important property of the boundedness of neural activities can be guaranteed by global inhibition. If used together with self-excitation, the global inhibition gives rise to a multi stable winner-take-all (WTA) mechanism. A condition for a neuron to be a potential winner of the competing dynamics is derived. The network becomes a largest input selector when the self-excitation is marginal.Slowing down the global inhibition produces oscillations. The study of oscillations of random networks suggests that all cyclic trajectories of linear threshold networks are a result of the existence of partitions with undamped linear oscillations. Chaotic dynamics were never encountered in computer simulations and perhaps do not exist at all in small networks.

Subfields: Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

Maxout Networks (2013, background): “Note that the arguments in section 7 motivating the use of maxout also apply equally to rectiﬁed linear units (Salinas & Abbott, 1996; Hahnloser, 1998; Glorot et al., 2011).”
On the Convergence Rate of Training Recurrent Neural Networks (2018, background): “In fact, one of the earliest adoptions of ReLUs was on applications of RNNs for this purpose twenty years ago [21, 46].”
Backward Feature Correction: How Deep Learning Performs Deep Learning (2020, background): “(3)Indeed, since learning a 2-layer network with three hidden units is already NP-hard [27] due to the extreme non-convexity this, it could be challenging to show how such contrived concept class could be learnt by a even more non-convex multi-layer neural network efficiently from scratch.”
VAMPnets for deep learning of molecular kinetics (2017, methodology): “All hidden layers employ Rectiﬁed Linear Units (ReLU) [66, 67].”
Deep Sparse Rectifier Neural Networks (2011, background): “Most previous studies of neural networks involving a rectifying activation function concern recurrent networks (Salinas and Abbott, 1996; Hahnloser, 1998).”