Guessing can Outperform Many Long Time Lag Algorithms

By J. Urgen Schmidhuber, Sepp Hochreiter. 1996

Subfields: Optimization & Training, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

In Scaling & Learning Theory

Citations in the corpus

Adaptive Computation Time for Recurrent Neural Networks (2016, background): “1 Parity Determining the parity of a sequence of binary numbers is a trivial task for a recurrent neural network [27], which simply needs to implement an internal switch that changes sign every time a one is received.”
Long Short-Term Memory (1997, unknown): “Recently we discovered (Schmidhuber & Hochreiter, 1996; Hochreiter & Schmidhuber, 1996, 1997) that many long-time-lag tasks used in previous work can be solved more quickly by simple random weight guessing than by the proposed algorithms.”
The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions (1998, unknown): “With the value := nwmax 4:0 < 1:0 we get the exponential decay @#v (t q) @#u (t) n ( )q : (18) 2.”
Neuroevolution strategies for episodic reinforcement learning (2009, background): “Random weight guessing (RWG) repeatedly draws all weights of the neural network randomly from fixed, a priori fixed intervals, evaluates the corresponding policy, and keeps the best policy found so far [39].”
Bridging Long Time Lags by Weight Guessing and \Long Short Term Memory" (1996, background): “They can be solved more quickly by random weight guessing than by the proposed algorithms [27].”
The Sparseness of Mixed Selectivity Neurons Controls the Generalization-Discrimination Trade-Off (2013, background): “The importance of this component of the algorithms is often underestimated (Schmidhuber and Hochreiter, 1996).”