Improving Generalization for Temporal Difference Learning: The Successor Representation

By Peter S. Dayan. 1993

Subfields: Reinforcement Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Reinforcement Learning with Unsupervised Auxiliary Tasks (2016, background): “Similarly, the successor representation (Dayan, 1993; Barreto et al., 2016; Kulkarni et al., 2016) factors a continuous set of expected value functions for a fixed policy, by combining an expectation over features of the state with an embedding of the pseudo-reward function.”
Weakly-Supervised Disentanglement Without Compromises (2020, background): “samples from a distribution (Dayan, 1993; Storck et al., 1995; Hochreiter & Schmidhuber, 1999; Bengio et al., 2013; Peters et al., 2017; Thomas et al., 2017; Schölkopf, 2019).”
Shaping Belief States with Generative Environment Models for RL (2019, background): “A non-exhaustive list includes pixel control [55], contrastive predictive coding (CPC) [56], action-conditional CPC [20], frame reconstruction [35, 57], next-frame prediction [18, 58, 20, 59] and successor representations [60, 61, 62].”
Independently Controllable Features (2017, background): “There are many more works that go in this direction, such as (deep) successor feature representations (Dayan, 1993; Kulkarni et al., 2016) or the options framework (Sutton et al., 1999; Precup, 2000) when used in conjunction with neural networks (Bacon et al., 2016).”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “We refer readers to Dayan (1993), Kulkarni et al. (2016), Barreto et al. (2017) and Zhang et al. (2017a) for more detailed discussions and extensions.”
Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation (2016, background): “Decomposition of the successor representation have shown to yield reasonable subgoals for spatial navigation problems [5, 30].”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Moreover, recent work has demonstrated that the successor representation model from the RL literature (Dayan, 1993) may provide a more accurate model to reflect the way humans balance their use of both model-based and model-free learning (Momennejad, Russek, Cheong, Botvinick, Daw, and Gershman, 201”
Machine Theory of Mind (2018, background): “…about future behaviour, such as next-step action probabilities ( ˆ π ), probabilities of whether certain objects will be consumed ( ˆ c ), and predicted successor representations ( (cid:99) SR ; Dayan, 1993). acter and mental state embeddings to predict subsequent behaviour of the agent.”
FeUdal Networks for Hierarchical Reinforcement Learning (2017, neutral): “…a long standing topic in re-inforcement learning (Sutton et al., 1999; Precup, 2000; Dayan & Hinton, 1993; Dietterich, 2000; Boutilier et al., 1997; Dayan, 1993; Kaelbling, 2014; Parr & Russell, 1998; Precup et al., 1997; 1998; Schmidhuber, 1991; Sutton, 1995; Wiering & Schmidhuber, 1997;…”
Successor Features for Transfer in Reinforcement Learning (2016, methodology): “This is essentially the concept of SR extended from the space S to the set S ×A× S [7].”
Deep Successor Reinforcement Learning (2016, background): “2), we can express the SR in a recursive form: Given the SR, the Q-value for selecting action a in state s can be expressed as the inner product of the immediate reward and the SR [5]:”
Disentangling the independently controllable factors of variation by interacting with the world (2018, background): “The idea that learning good representations is vital for solving most kinds of real-world problems is not new, both in the supervised learning literature [Bengio, 2009, Goodfellow et al., 2016], and in the RL literature [Dayan, 1993, Precup, 2000].”
A Brief Survey of Deep Reinforcement Learning (2017, neutral): “A less common but potentially useful paradigm exists between model-free and model-based methods—the successor representation (SR) [24].”
Strategic Attentive Writer for Learning Macro-Actions (2016, background): “2 Related Work Learning temporally extended actions and temporal abstraction in general are long standing problems in reinforcement learning [5, 6, 7, 8, 12, 20, 21, 22, 23, 25, 27, 28, 30].”
Visual Semantic Planning Using Deep Successor Representations (2017, methodology): “Successor representation (SR), proposed by Dayan [7], can be considered as a hybrid approach of model-based and model-free RL. Barreto et al. [3] derived a bound on value functions of an optimal policy when transferring policies using successor representations.”
Reinforcement Learning with Replacing Eligibility Traces (2005, background): “Arguably, yet a third mechanism for managing delayed reward is to change representations orworld models (e.g., Dayan, 1993 ; Sutton, 1995).2.”
Learning by Playing - Solving Sparse Reward Tasks from Scratch (2018, background): “These are in turn are inherently connected to learning to predict the future via “successor” representations (Dayan, 1993; Kulka-rni et al., 2016b; Barreto et al., 2017) or forecasts (Schaul & Ring, 2013; Lample & Chaplot, 2017; Dosovitskiy & Koltun, 2017) and are trained to be predictive of…”
Deep reinforcement learning with successor features for navigation across similar environments (2016, methodology): “[25] P. Dayan, “Improving generalization for temporal difference learning: The successor representation,” Neural Computation, vol. 5, no. 4, 1993.”