Untersuchungen zu dynamischen neuronalen Netzen

By Sepp Hochreiter. 1991

Subfields: Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

In Optimization & Training

Citations in the corpus

Deep Residual Learning for Image Recognition (2015, background): “Driven by the significance of depth, a question arises: Is learning better networks as easy as stacking more layers? An obstacle to answering this question was the notorious problem of vanishing/exploding gradients [14, 1, 8], which hamper convergence from the beginning.”
Long Short-Term Memory-Networks for Machine Reading (2016, background): “The first one concerns model training problems associated with vanishing and exploding gradients (Hochreiter, 1991; Bengio et al., 1994), which can be partially ameliorated with gated activation functions, such as the Long ShortTerm Memory (LSTM) (Hochreiter and Schmidhuber, 1997), and gradient clip”
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling (2014, background): “These shortcuts allow the error to be back-propagated easily without too quickly vanishing (if the gating unit is nearly saturated at 1 ) as a result of passing through multiple, bounded nonlinearities, thus reducing the difﬁculty due to vanishing gradients [Hochreiter, 1991, Bengio et al., 1994].”
Sequence to Sequence Learning with Neural Networks (2014, background): “While it could work in principle since the RNN is provided with all the relevant information, it would be difficult to train the RNNs due to the resulting long term dependencies (figure 1) [14, 4, 16, 15].”
Neural Machine Translation by Jointly Learning to Align and Translate (2014, background): “These paths allow gradients to ﬂow backward easily without suffering too much from the vanishing effect (Hochreiter, 1991; Bengio et al. , 1994; Pascanu et al. , 2013a).”
Training Very Deep Networks (2015, background): “Very deep network training still faces problems, albeit perhaps less fundamental ones than the problem of vanishing gradients in standard recurrent networks [28].”
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks (2013, background): “These successes have been achieved despite the noted difficulty of training such deep architectures [7, 8, 9, 10, 11].”
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) (2015, background): “Saturation means a small derivative of the activation function which decreases the variation and the information that is propagated to the next layer Hochreiter et al. (2001b); Hochreiter (1997; 1998). ELUs code the degree of presence of input concepts, while they do not quantify the degree of their”
The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation (2018, background): “In practice, RNNs are notoriously hard to train (Hochreiter, 1991; Bengio et al., 1994; Hochreiter et al., 2001), confirming the well”
Neural Approaches to Conversational AI (2018, positive): “This model introduces a temporal structure that makes the hidden state of the current dialogue turn directly dependent on the hidden state of the previous dialogue turn, effectively allowing information to flow over longer time spans, and helping reduce the vanishing gradient problem (Hochreiter, 19”
Recurrent Highway Networks (2016, background): “Deep networks suffer from what are commonly referred to as the vanishing and exploding gradient problems (Hochreiter, 1991; Bengio et al., 1994; Hochreiter et al., 2001), since the magnitude of the gradients may shrink or explode exponentially during backpropagation.”
Grid Long Short-Term Memory (2015, background): “…suffer from exactly the same problems as recurrent networks applied to long sequences: namely that information from past computations rapidly attenuates as it progresses through the chain – the vanishing gradient problem [18] – and that each layer cannot dynamically select or ignore its inputs.”
LSTM: A Search Space Odyssey (2015, background): “Since LSTMs are effective at capturing long-term temporal dependencies without suffering from the optimization hurdles that plague simple recurrent networks (SRNs) (Hochreiter, 1991; Bengio et al., 1994), they have been used to advance the state of the art for many difﬁcult problems.”
Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses (2017, methodology): “LSTMs are one of the most well-established methods for dealing with the vanishing gradient problem in recurrent networks (Hochreiter, 1991; Bengio et al., 1994).”
Recurrent Batch Normalization (2016, background): “However, training RNNs using ﬁrst-order stochastic gradient descent (SGD) is notoriously difﬁcult due to the well-known problem of exploding/vanishing gradients (Bengio et al., 1994; Hochreiter, 1991; Pascanu et al., 2012).”
Gated Feedback Recurrent Neural Networks (2015, background): “A recurrent neural network (RNN) is able to process a sequence of arbitrary length by recursively applying a transition function to its internal hidden state for each symbol of the input sequence.”
A Clockwork RNN (2014, background): “Standard RNNs trained with stochastic gradient descent have difﬁculty learning long-term dependencies (i.e. spanning more that 10 time-steps) encoded in the input sequences due to the vanishing gradient (Hochreiter, 1991; Hochre-iter et al., 2001).”
Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations (2016, background): “This helps combat the vanishing gradient problem (Hochreiter, 1991; Bengio et al., 1994), as we observe experimentally.”
Deep, Big, Simple Neural Nets for Handwritten Digit Recognition (2010, background): “Training them is hard, as backpropagated gradients quickly vanish exponentially in the number of layers (Hochreiter, 1991; Hochreiter, Bengio, Frasconi, & Schmidhuber, 2001; Hinton, 2007), just as the first recurrent neural networks (Hochreiter & Schmidhuber, 1997).”
Deep learning in neural networks: An overview (2014, background): “9 1991: Fundamental Deep Learning Problem of Gradient Descent A diploma thesis (Hochreiter, 1991) represented a milestone of explicit DL research.”