Sparse Attentive Backtracking: Long-Range Credit Assignment in Recurrent Networks
By Nan Rosemary Ke, Anirudh Goyal, Olexa Bilaniuk, Jonathan Binas, Laurent Charlin, C. Pal, Yoshua Bengio. 2017
15 citations recorded by the source corpus.
A major drawback of backpropagation through time (BPTT) is the difficulty of learning long-term dependencies, coming from having to propagate credit information backwards through every single step of the forward computation. This makes BPTT both computationally impractical and biologically implausible. For this reason, full backpropagation through time is rarely used on long sequences, and truncated backpropagation through time is used as a heuristic. However, this usually leads to biased estimates of the gradient in which longer term dependencies are ignored. Addressing this issue, we propose an alternative algorithm, Sparse Attentive Backtracking, which might also be related to principles used by brains to learn long-term dependencies. Sparse Attentive Backtracking learns an attention mechanism over the hidden states of the past and selectively backpropagates through paths with high attention weights. This allows the model to learn long term dependencies while only backtracking for a small number of time steps, not just from the recent past but also from attended relevant past states.
Subfields: Artificial Intelligence, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Ruslan Salakhutdinov
- Mohit Bansal
- Pieter Abbeel
- Percy Liang
- Quoc V. Le
- Noah Constant
- Charles Blundell
In Reinforcement Learning
- Bilal Piot
- Charles Blundell
- Adrià Puigdomènech Badia
- Steven Kapturowski
- Pablo Sprechmann
- Rami Al-Rfou
- Alex Vitvitskyi
- Daniel Guo
Citations in the corpus
- Character-Level Language Modeling with Deeper Self-Attention (2018, methodology): “A range of methods have arisen for unbiasing and improving TBTT (Tallec and Ollivier 2017; Ke et al. 2017).”
- Agent57: Outperforming the Atari Human Benchmark (2020, background): “Recent work on long-term credit assignment can be categorized into roughly two types: ensuring that gradients correctly assign credit (Ke et al., 2017; Weber et al., 2019; Fortunato et al., 2019) and using values or targets to ensure correct credit is assigned (Arjona-Medina et al., 2019; Hung et…”