Orthogonal RNNs and Long-Memory Tasks
By Mikael Henaff, Arthur Szlam, Yann LeCun. 2016
44 citations recorded by the source corpus.
Subfields: Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Model Architectures
- Yoshua Bengio
- Quoc V. Le
- David R Ha
- C. Pal
- J. Urgen Schmidhuber
- Andrew M. Dai
- Yann LeCun
- Aaron C. Courville
Citations in the corpus
- HyperNetworks (2016, methodology): “Unlike previous approaches (Graves, 2013; Ognawala & Bayer, 2014) of applying weight noise during training, we instead also apply dropout to the recurrent layer (Henaff et al., 2016) with the same dropout probability.”
- On orthogonality and learning recurrent networks with long term dependencies (2017, methodology): “Le et al. (2015) and Henaff et al. (2016) have respectively shown that identity initialization and orthogonal initialization can be ben-eficial.”
- Dynamic Layer Normalization for Adaptive Neural Acoustic Modeling in Speech Recognition (2017, background): “All weights are initialized by orthogonal initialization [25] and biases are set to zero.”
- Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs (2016, methodology): “Specifically, the task is defined as follows (Hochreiter & Schmidhuber, 1997; Arjovsky et al., 2015; Henaff et al., 2016).”
- Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflections (2016, methodology): “In addition to these initialisation methods, one study also considered removing the non-linearity between the hidden-to-hidden connections (Henaff et al., 2016), i.”
- Revisiting NARX Recurrent Neural Networks for Long-Term Dependencies (2017, methodology): “…Hihi & Bengio, 1995; Koutnik et al., 2014), using associative or explicit memory (Plate, 1993; Danihelka et al., 2016; Graves et al., 2014; Weston et al., 2015; Sukhbaatar et al., 2015), and initializing or restricting weight matrices to be orthogonal (Arjovsky et al., 2016; Henaff et al., 2016).”
- Task-Oriented Feature Distillation (2020, background): “2 Orthogonal Loss As a desirable property in both convolutional neural networks and recurrent neural networks, the orthogonality of weights can solve the vanishing and exploding gradients problem by stabilizing the norm of gradients [61, 4, 19].”