Bridging Long Time Lags by Weight Guessing and \Long Short Term Memory"

By Sepp Hochreiter. 1996

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

Training Very Deep Networks (2015, background): “But the natural solution for arbitrary n is a recurrent net with only 3 units and 5 weights, reading the input bit string one bit at a time, making a single recurrent hidden unit ﬂip its state whenever a new 1 is observed [7].”
Deep learning in neural networks: An overview (2014, background): “In fact, it turned out that certain simple but deep benchmark problems used to evaluate such methods are more quickly solved by randomly guessing RNN weights until a solution is found (Hochreiter & Schmidhuber, 1996).”
The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions (1998, unknown): “The vector @Y (t) @netv(t q) is de ned as @Y (t) @netv (t q) i := @yi (t) @netv (t q) (10) for q 0 and the matrix rY (t 1)Y (t) is de ned as rY (t 1)Y (t) ij := @yi (t) @yj (t 1) : (11) From the de nitions rY (t 1)Y (t) = F 0 (t) W (12)”
A Critical Review of Recurrent Neural Networks for Sequence Learning (2015, background): “Figure 9: One LSTM memory cell as proposed by Hochreiter and Schmidhuber [1997]. The self-connected node is the internal state s.”
Long Short-Term Memory (1997, unknown): “, Pollack (1991). But a real long time lag problem does not have any short time lag exemplars in the training set.”
Learning Recurrent Neural Networks with Hessian-Free Optimization (2011, methodology): “However, none of these methods became widely adopted, and an analysis by Hochreiter and Schmidhuber (1996) showed that they were often no better than random guessing.”