Hierarchical Recurrent Neural Networks for Long-Term Dependencies

By Salah El Hihi, Yoshua Bengio. 1995

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation (2013, background): “Multi-scale temporal hierarchies for recurrent nets have already been proposed and involved exponential moving averages of different time constants (El Hihi and Bengio, 1996), where each unit is still updated after each time step.”
The challenge of realistic music generation: modelling raw audio at scale (2018, background): “Other means of enabling neural networks to learn long-range dependencies in data that have been investigated include alternative loss functions [42] and model architectures [1, 8, 14, 19, 28, 29, 32].”
Listen, attend and spell: A neural network for large vocabulary conversational speech recognition (2015, background): “We provide more details of these components in the following sections.”
How to Construct Deep Recurrent Neural Networks (2013, background): “An RNN may be extended deeper in yet another way by stacking multiple recurrent hidden layers on top of each other (Schmidhuber, 1992; El Hihi and Bengio, 1996; Jaeger, 2007; Graves, 2013).”
Gated Feedback Recurrent Neural Networks (2015, background): “A conventional way to encode this hierarchy in an RNN has been to stack multiple levels of recurrent layers (Schmidhuber, 1992; El Hihi & Bengio, 1995; Graves, 2013; Hermans & Schrauwen, 2013).”
Deep learning in neural networks: An overview (2014, background): “…(Bodenhausen & Waibel, 1991), Nonlinear AutoRegressive with eXogenous inputs (NARX) RNNs (Lin, Horne, Tino, & Giles, 1996), certain hierarchical RNNs (Hihi & Bengio, 1996) (compare Section 5.10, 1991), RL economies in RNNs with WTA units and local learning rules (Schmidhuber, 1989b), and other…”
Recurrent Independent Mechanisms (2019, background): “Computation on demand: There are various architectures (El Hihi & Bengio, 1996; Koutnik et al., 2014; Chung et al., 2016; Neil et al., 2016; Jernite et al., 2016; Krueger et al., 2016) where parts of the LSTM’s hidden state are kept dormant at times.”
An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling (2018, background): “Many other architectural innovations and training techniques for recurrent networks have been introduced and continue to be actively explored (El Hihi & Bengio, 1995; Schuster & Paliwal, 1997; Gers et al., 2002; Koutnik et al., 2014; Le et al., 2015; Ba et al., 2016; Wu et al., 2016; Krueger et…”
Dilated Recurrent Neural Networks (2017, background): “The dilated skip connection can be represented as This is similar to the regular skip connection[8, 30], which can be represented as s ( l ) is referred to as the skip length, or dilation of layer l ; x ( l ) t as the input to layer l at time t ; and f ( · ) denotes any RNN cell and output…”
Deep visual foresight for planning robot motion (2016, background): “Of particular interest for robotic manipulation, hierarchical models operating at varying time scales [43] may even make it practical to plan highly elaborate skills entirely using learned predictive models.”
A Hierarchical Recurrent Encoder-Decoder for Generative Context-Aware Query Suggestion (2015, background): “The idea of encoding hierarchical multi-scale representations is also explored in [15].”
Very deep convolutional networks for end-to-end speech recognition (2016, methodology): “We show how to apply NiN principles in hierarchical Recurrent Neural Networks (RNNs) [20].”