Recurrent Neural Net Learning and Vanishing

By Sepp Hochreiter. 1998

Subfields: Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

1. Grgoire Montavon
2. Thomas Unterthiner
3. Sepp Hochreiter
4. K. Müller
5. Wojciech Samek
6. Kyunghyun Cho
7. Yoshua Bengio
8. Peter Sadowski

In Optimization & Training

1. Thomas Unterthiner
2. Pierre Baldi
3. Peter Sadowski
4. Sepp Hochreiter
5. Grgoire Montavon
6. Wojciech Samek
7. Daniel Whiteson
8. K. Müller

Citations in the corpus

• RUDDER: Return Decomposition for Delayed Rewards (2018, background): “Uniform Credit Assignment The LSTM network has been proposed with the aim to learn long-term dependencies in sequences which span over long intervals [47, 48, 42, 43].”
• Searching for exotic particles in high-energy physics with deep learning (2014, methodology): “Recent successes in deep learning—for example, neural networks with multiple hidden layers—have come from alleviating the gradient diffusion problem by a combination of factors, including: (1) speeding up the stochastic gradient descent algorithm with graphics processors; (2) using much larger train”
• Deep Learning, Dark Knowledge, and Dark Matter (2014, background): “Historically, these networks were considered difficult to train, in part due to the vanishing gradient problem (Bengio et al., 1994; Hochreiter, 1998; Larochelle et al., 2009), but advances primarily in computing power and available training data have made these methods significantly more practical.”
• Explaining and Interpreting LSTMs (2019, background): “The LSTM network has been proposed with the aim to learn long-term dependencies in sequences which span over long intervals [37, 36, 31, 32].”