Deep Successor Reinforcement Learning

By Tejas D. Kulkarni, Ardavan Saeedi, Simanta Gautam, Samuel J. Gershman. 2016

232 citations recorded by the source corpus.

Learning robust value functions given raw observations and rewards is now possible with model-free and model-based deep reinforcement learning algorithms. There is a third alternative, called Successor Representations (SR), which decomposes the value function into two components -- a reward predictor and a successor map. The successor map represents the expected future state occupancy from any given state and the reward predictor maps states to scalar rewards. The value function of a state can be computed as the inner product between the successor map and the reward weights. In this paper, we present DSR, which generalizes SR within an end-to-end deep reinforcement learning framework. DSR has several appealing properties including: increased sensitivity to distal reward changes due to factorization of reward and world dynamics, and the ability to extract bottleneck states (subgoals) given successor maps trained under a random policy. We show the efficacy of our approach on two diverse environments given raw pixel observations -- simple grid-world domains (MazeBase) and the Doom game engine.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments (2017, background): “RL in navigation A number of recent papers use reinforcement learning (RL) to train navigational agents [25, 41, 44, 51], although these works do not address language instruction, which is the focus of this paper.”
Rainbow: Combining Improvements in Deep Reinforcement Learning (2017, methodology): “Among successful applications of HRL we highlight h-DQN (Kulkarni et al. 2016a) and Feu-dal Networks (Vezhnevets et al. 2017).”
Value Prediction Network (2017, background): “A lot of variations of DQN have been proposed for learning better state representation [37, 16, 9, 22, 36, 24], including the use of memory-based networks for handling partial observability [9, 22, 24], estimating both state-values and advantage-values as a decomposition of Q-values [37], learning s”
Shaping Belief States with Generative Environment Models for RL (2019, background): “A non-exhaustive list includes pixel control [55], contrastive predictive coding (CPC) [56], action-conditional CPC [20], frame reconstruction [35, 57], next-frame prediction [18, 58, 20, 59] and successor representations [60, 61, 62].”
Independently Controllable Features (2017, background): “There are many more works that go in this direction, such as (deep) successor feature representations (Dayan, 1993; Kulkarni et al., 2016) or the options framework (Sutton et al., 1999; Precup, 2000) when used in conjunction with neural networks (Bacon et al., 2016).”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “We refer readers to Dayan (1993), Kulkarni et al. (2016), Barreto et al. (2017) and Zhang et al. (2017a) for more detailed discussions and extensions.”
Learning to Act by Predicting the Future (2016, methodology): “We have compared the presented approach to three deep RL methods: DQN (Mnih et al., 2015), A3C (Mnih et al., 2016), and DSR (Kulkarni et al., 2016b).”
Learning to Navigate in Complex Environments (2016, methodology): “iguation in partially observable environments (Koutnk et al., 2013; Hausknecht & Stone, 2015; Mnih et al., 2016; Narasimhan et al., 2015). Deep RL has recently been used in the navigation domain. Kulkarni et al. (2016) used a feedforward architecture to learn deep successor representations that ”
Gated-Attention Architectures for Task-Oriented Language Grounding (2017, methodology): “Prior work has explored using Deep Re-inforcement learning approaches for playing FPS games (Lample and Chaplot, 2016; Kempka et al., 2016; Kulkarni et al., 2016).”
A Brief Survey of Deep Reinforcement Learning (2017, neutral): “also an important area of ongoing research [117, 66, 152].”
Visual Semantic Planning Using Deep Successor Representations (2017, methodology): “[24] proposed a method to learn deep successor features.”
Learning by Playing - Solving Sparse Reward Tasks from Scratch (2018, methodology): “…Reinforcement Learning with the help of so called “intrinsic motivation” rewards (Chentanez et al., 2005; Singh et al., 2009) has, furthermore, been studied for controlling real robots by Ngo et al. (2012) and combined with Deep RL techniques by Kulkarni et al. (2016a); Dilokthanakul et al. (2017).”
Deep reinforcement learning with successor features for navigation across similar environments (2016, methodology): “challenging control tasks [5], [6], [7], [8], [9].”
Independently Controllable Factors (2017, background): “Many more works go in this direction, such as (deep) successor feature representations (Dayan, 1993; Kulkarni et al., 2016) or the options framework (Sutton et al.”
γ-Models: Generative Temporal Difference Learning for Infinite-Horizon Prediction (2020, background): “The most closely related prior work to γ -models is the successor representation (Dayan, 1993), a formulation of long-horizon prediction that has been inﬂuential in both cognitive science (Momenne-jad et al., 2017; Gershman, 2018) and machine learning (Kulkarni et al., 2016; Ma et al., 2018).”
Transfer Learning in Deep Reinforcement Learning: A Survey (2020, methodology): “Specifically, the work of [127] was developed based on a weaker”
Automated Curriculum Learning by Rewarding Temporally Rare Events (2018, methodology): “Several deep RL approaches have been applied to Doom, which include auxiliary learning [21, 22],”
Task-Agnostic Dynamics Priors for Deep Reinforcement Learning (2019, background): “Successor representations (Dayan, 1993) decompose the agent’s value function into a feature-based state representation and a reward projection operator, resulting in better exploration of the state space (Kulkarni et al., 2016; Bar-reto et al., 2017; Machado et al., 2017b).”