Recurrent Orthogonal Networks and Long-Memory Tasks

By Mikael Henaff, Arthur Szlam, Yann LeCun. 2016

147 citations recorded by the source corpus.

Although RNNs have been shown to be powerful tools for processing sequential data, finding architectures or optimization strategies that allow them to model very long term dependencies is still an active area of research. In this work, we carefully analyze two synthetic datasets originally outlined in (Hochreiter and Schmidhuber, 1997) which are used to evaluate the ability of RNNs to store information over many time steps. We explicitly construct RNN solutions to these problems, and using these constructions, illuminate both the problems themselves and the way in which RNNs store different types of information in their hidden states. These constructions furthermore explain the success of recent methods that specify unitary initializations or constraints on the transition matrices.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

The challenge of realistic music generation: modelling raw audio at scale (2018, background): “Other means of enabling neural networks to learn long-range dependencies in data that have been investigated include alternative loss functions [42] and model architectures [1, 8, 14, 19, 28, 29, 32].”
On orthogonality and learning recurrent networks with long term dependencies (2017, methodology): “Furthermore, Henaff et al. (2016) derive a solution mechanism for the copy task that drops the non-linearity from an RNN.”
How to Start Training: The Effect of Initialization and Architecture (2018, background): “Activation lengths can also be controlled by constraining weight matrices to be orthogonal or unitary (see e.g. [1, 10, 13, 18, 24]).”
HiPPO: Recurrent Memory with Optimal Polynomial Projections (2020, background): “A long line of work controls the spectrum of the recurrent updates with (nearly-) orthogonal matrices to control gradients [3], but have been found to be less robust across different tasks [33].”
Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflections (2016, methodology): “In addition to these initialisation methods, one study also considered removing the non-linearity between the hidden-to-hidden connections (Henaff et al., 2016), i.e. the term Wht−1 in Equation (1) is outside the activation function φ.”
Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group (2019, methodology): “For the initialization of the layer with a matrix A 0 ∈ Skew( p ) , we drew ideas from both (Henaff et al., 2016) and (Helfrich et al., 2018).”
Initialization matters: Orthogonal Predictive State Recurrent Neural Networks (2018, background): “Related results are presented in Henaff et al. (2016), Saxe et al. (2013) (orthogonal initialization for training acceleration), Ganguli et al. (2008) and White et al. (2004).”
Residual Connections Encourage Iterative Inference (2017, background): “This suggests that future unrolled Resnets should use insights from research on recurrent networks optimization, including careful initialization (Henaff et al., 2016) and parametrization changes (Hochreiter & Schmidhuber, 1997).”
Continual Learning with Gated Incremental Memories for sequential data processing (2020, background): “It is well known that orthogonal initialization of memory weight matrixes can improve learning when dealing with long sequences and linear memories [27, 35].”
Long Expressive Memory for Sequence Modeling (2021, background): “An alternative approach imposes constraints on the structure of the hidden weight matrices of the underlying recurrent neural networks (RNNs), for instance by requiring these matrices to be unitary or orthogonal (Henaff et al., 2016; Arjovsky et al., 2016; Wis-dom et al., 2016; Kerg et al., 2019).”
Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients? (2018, methodology): “…to control gradients; (ii) precisely initializing weights (e.g. i.i.d. with properly chosen variances [MM15, HZRS15] or using orthogonal weight matrices [ASB16, HSL16]); (iii) choosing non-linearities that that tend to compute numerically stable gradients or activations at initialization [KUMH17].”
Orthogonal Over-Parameterized Training (2020, background): “It is essentially conducting gradient descent on the Stiefel manifold [44, 74, 75, 42, 3, 22, 33].”
Learning Longer-term Dependencies in RNNs with Auxiliary Losses (2018, background): “…2014; Le et al., 2015; He et al., 2016), highway connections (Zilly et al., 2017), orthogonal-or unitary-constrained weights (White et al., 2004; Henaff et al., 2016; Arjovsky et al., 2016), dilated convolu-tions (Salimans et al., 2017), connections (Koutnik et al., 2014) and attention…”
Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence (2019, background): “At the other end of the spectrum, there is complementary work addressing initialization of deep networks, [10, 13].”