A Fast Stochastic Error-Descent Algorithm for Supervised Learning and Optimization

By Gert Cauwenberghs. 1992

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Deep learning in neural networks: An overview (2014, methodology): “…1970) are computationally too expensive for large NNs. Partial BFGS (Battiti, 1992; Saito & Nakano, 1997) and conjugate gradient (Hestenes & Stiefel, 1952; Møller, 1993) as well as other methods (Cauwenberghs, 1993; Schmidhuber, 1989a; Solla, 1988) provide sometimes useful fast alternatives.”
A reward-modulated Hebbian learning rule can explain experimentally observed network reorganization in a brain control task (2010, background): “Either the tunable parameters of the system (weights) are perturbed (Jabri and Flower, 1992; Cauwenberghs, 1993; Seung, 2003) or the output of nodes in the network are perturbed (Mazzoni et al.”
An analysis of noise in recurrent neural networks: convergence and generalization (1996, background): “Recently, stochastic techniques, such as Nodal Perturbation, Weight Perturbation, and Summed Weight Neuron Perturbation, have been introduced which in some cases may converge more quickly than pure gradient descent (Cauwenberghs [7], Dembo and Kailath [9], Flower and Jabri [10, 17]).”
Coherent Beam Combining of Fiber Amplifiers Using Stochastic Parallel Gradient Descent Algorithm and Its Application (2009, unknown): “reasoning, not strict mathematical deduction [32].”
Learning in neural networks by reinforcement of irregular spiking. (2004, background): “Therefore, similar to other types of reinforcement learning [2,3,15], the learning rule suffers from slow convergence in large networks.”
Fast Exact Multiplication by the Hessian (1994, unknown): “In weight perturbation (Jabri and Flower 1991; Alspector et al. 1993; Flower and Jabri 1993; Kirk et al. 1993; Cauwenberghs 1993) the gradient 0, is approximated using only the globally broadcast result of the computation of E(w).”
Learning curves for stochastic gradient descent in linear feedforward networks (background): “The simplest case is when noise is added directly to the weight matrix: E WP = 1 2 |(W + ψ)x| Such an approach is sometimes termed 'weight perturbation' [2] [4].”
Learning in spiking neural networks by reinforcement of stochastic synaptic transmission. (2003, background): “Thisisbecause C ( (cid:6) )vanisheswhen (cid:6) becomesmuchlonger than the mixing time of the Markov chain, but the fluctuations in and Flower, 1992; Cauwenberghs, 1993; Unnikrishnan and Venugo-Neuron1072”
Experimentally realized in situ backpropagation for deep learning in photonic neural networks (2022, background): “[47] Gert Cauwenberghs, “A Fast Stochastic Error-Descent Algorithm for Supervised Learning and Optimization,” in Advances in Neural Information Processing Systems 5 (1992).”
A theory for how sensorimotor skills are learned and retained in noisy and nonstationary neural circuits (2013, background): “Here we note that examples of the exploratory benefits of noise are found in the concepts of simulated annealing (20), reinforcement learning (21), and other stochastic learning methods (22).”