Learning Longer-term Dependencies in RNNs with Auxiliary Losses

By Trieu H. Trinh, Andrew M. Dai, Thang Luong, Quoc V. Le. 2018

200 citations recorded by the source corpus.

Despite recent advances in training recurrent neural networks (RNNs), capturing long-term dependencies in sequences remains a fundamental challenge. Most approaches use backpropagation through time (BPTT), which is difficult to scale to very long sequences. This paper proposes a simple method that improves the ability to capture long term dependencies in RNNs by adding an unsupervised auxiliary loss to the original objective. This auxiliary loss forces RNNs to either reconstruct previous events or predict next events in a sequence, making truncated backpropagation feasible for long sequences and also improving full BPTT. We evaluate our method on a variety of settings, including pixel-by-pixel image classification with sequence lengths up to 16\,000, and a real document classification benchmark. Our results highlight good performance and resource efficiency of this approach over competitive baselines, including other recurrent models and a comparable sized Transformer. Further analyses reveal beneficial effects of the auxiliary loss on optimization and regularization, as well as extreme cases where there is little to no backpropagation.

Subfields: Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, background): “…many efforts have been spent on relieving the vanishing gradient problem, including better initialization (Le et al., 2015), additional loss signal (Trinh et al., 2018), augmented memory structure (Ke et al., 2018) and others that modify the internal architecture of RNNs to ease the optimization…”
The challenge of realistic music generation: modelling raw audio at scale (2018, background): “Other means of enabling neural networks to learn long-range dependencies in data that have been investigated include alternative loss functions [42] and model architectures [1, 8, 14, 19, 28, 29, 32].”
Efficiently Modeling Long Sequences with Structured State Spaces (2021, neutral): “[4] 99.20 98.13 73.42 TCN [3] 99.0 97.2 - LSTM [17, 21] 98.9 95.11 63.01 r-LSTM [42] 98.4 95.2 72.2 Dilated GRU [5] 99.0 94.6 - Dilated RNN [5] 98.0 96.1 - IndRNN [25] 99.0 96.0 - expRNN [24] 98 Baselines.”
HiPPO: Recurrent Memory with Optimal Polynomial Projections (2020, background): “[69] Trieu H Trinh, Andrew M Dai, Minh-Thang Luong, and Quoc V Le.”
Kaleidoscope: An Efficient, Learnable Representation For All Structured Linear Maps (2020, background): “Many new RNN architectures, with unitary or orthogonal weight matrices to avoid gradient explosion or vanishing, have been proposed and tested on this task (Le et al., 2015; Arjovsky et al., 2016; Wisdom et al., 2016; Mhammedi et al., 2017; Trinh et al., 2018).”
Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs (2019, background): “Another recently proposed approach to ensuring signal propagation in long sequence tasks introduces auxiliary loss functions (Trinh et al., 2018).”
Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting (2020, methodology): “Instead of modifying inputs, Truncated BPTT (Aicher, Foti, and Fox 2019) only uses last time steps to estimate the gradients in weight updates, and Auxiliary Losses (Trinh et al. 2018) enhance the gradients ﬂow by adding auxiliary gradients.”
Memory Augmented Recursive Neural Networks (2019, background): “There has also been an attempt to improve the performance of recurrent neural networks (Trinh et al., 2018) Despite the amount of eﬀort spent on augmenting recurrent neural networks, to the best of our knowledge, there has been no attempt to enable an external memory to recursive networks which…”
Deep Learning Based Recommender System (2017, background): “If going deeper give favorable results, how do we train the deep architecture? If not, what is the reason behind this? A possibility is to look into auxiliary losses at dierent layers in similar spirit to [147] albeit hierarchically instead of sequentially.”
Simplified State Space Layers for Sequence Modeling (2022, background): “[56] Trieu Trinh, Andrew Dai, Thang Luong, and Quoc Le.”
Trellis Networks for Sequence Modeling (2018, methodology): “On stress tests, trellis networks outperform recent results achieved by recurrent networks and self-attention (Trinh et al., 2018).”
State-Denoised Recurrent Neural Networks (2018, background): “Finally, in a similar methodological spirit, Trinh et al. [2018] have proposed an auxillary loss function to support the training of RNNs.”
Transformer-XL: Language Modeling with Longer-Term Dependency (2018, background): “…many efforts have been spent on relieving the vanishing gradient problem, including better initialization (Le et al., 2015), additional loss signal (Trinh et al., 2018) and others that modify the internal architecture of RNNs to ease the optimization Mikolov et al. (2014); Koutnik et al. (2014);…”
Calibration, Entropy Rates, and Memory in Language Models (2019, background): “…& Zweig, 2012; Ji et al., 2015; Wang & Cho, 2016) or document-level topics (Wang et al., 2017) to using LSTM recurrent neural networks with careful initialization (Le et al., 2015), auxiliary loss signals (Trinh et al., 2018) or augmented memory structures (Grave et al., 2016; Ke et al., 2018).”
Scene Memory Transformer for Embodied Agents in Long-Horizon Tasks (2019, methodology): “Second, RNNs have optimization difficulties over long sequences [33, 41] in backpropagation through time (BPTT).”
Meta reinforcement learning as task inference (2019, methodology): “More generally, our method is related to learning with auxiliary losses which has been shown to improve the performance of memory-based architectures both in reinforcement learning (e.g. Wayne et al. (2018); Jaderberg et al. (2016)), and in sequence modelling (Trinh et al., 2018).”
Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers (2021, background): “[56] Trieu H Trinh, Andrew M Dai, Minh-Thang Luong, and Quoc V Le.”