Successor Features for Transfer in Reinforcement Learning

By André Barreto, Will Dabney, R´emi Munos, Jonathan J. Hunt, Tom Schaul, David Silver, Hado van Hasselt. 2016

725 citations recorded by the source corpus.

Transfer in reinforcement learning refers to the notion that generalization should occur not only within a task but also across tasks. We propose a transfer framework for the scenario where the reward function changes between tasks but the environment's dynamics remain the same. Our approach rests on two key ideas: "successor features", a value function representation that decouples the dynamics of the environment from the rewards, and "generalized policy improvement", a generalization of dynamic programming's policy improvement operation that considers a set of policies rather than a single one. Put together, the two ideas lead to an approach that integrates seamlessly within the reinforcement learning framework and allows the free exchange of information across tasks. The proposed method also provides performance guarantees for the transferred policy even before any learning has taken place. We derive two theorems that set our approach in firm theoretical ground and present experiments that show that it successfully promotes transfer in practice, significantly outperforming alternative methods in a sequence of navigation tasks and in the control of a simulated robotic arm.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Reinforcement Learning

Citations in the corpus

DeepMDP: Learning Continuous Latent Space Models for Representation Learning (2019, background): “There exists an extensive body of literature on exploiting the transition function structure for representation learning (Mahadevan & Maggioni, 2007; Barreto et al., 2017), but these works do not rely on the reward function.”
Reinforcement Learning with Unsupervised Auxiliary Tasks (2016, methodology): “Successor representations have been used to transfer representations from one pseudo-reward to another (Barreto et al., 2016) or to different scales of reward (Kulkarni et al.”
Learning World Graphs to Accelerate Hierarchical Reinforcement Learning (2019, neutral): “the context of RL is an important practice often leading to performance gain (Taylor & Stone, 2009; Barreto et al., 2017).”
Towards Deep Symbolic Reinforcement Learning (2016, background): “Each of these shortcomings is an active area of research in the DRL community, where data efﬁcient learning [8, 9], transfer learning [10, 11, 12, 13], planning [14, 15], and transparency [16] are all hot topics.”
RUDDER: Return Decomposition for Delayed Rewards (2018, background): “ﬁc, therefore can in general not achieve expected future rewards being zero [28]. Successor features decouple environment and policy from rewards, but changing the reward changes the optimal policies [7, 6]. Temporal Value Transport (TVT) uses an attentional memory mechanism to learn a value functio”
Shaping Belief States with Generative Environment Models for RL (2019, background): “A non-exhaustive list includes pixel control [55], contrastive predictive coding (CPC) [56], action-conditional CPC [20], frame reconstruction [35, 57], next-frame prediction [18, 58, 20, 59] and successor representations [60, 61, 62].”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “We refer readers to Dayan (1993), Kulkarni et al. (2016), Barreto et al. (2017) and Zhang et al. (2017a) for more detailed discussions and extensions.”
Same State, Different Task: Continual Reinforcement Learning without Interference (2021, background): “A number of previous works have studied transfer in multitask RL settings where the goals within an environment change (Barreto et al. 2016; Schaul et al. 2015).”
A Survey of Embodied AI: From Simulators to Research Tasks (2021, background): “This is inspired by the attention mechanism in neural networks, and successor representations [89, 90] and features [91] in reinforcement learning.”
Composable Action-Conditioned Predictors: Flexible Off-Policy Learning for Robot Navigation (2018, background): “Multi-task methods often aim to capture multiple tasks within the same policy or value function [2, 3].”
DARLA: Improving Zero-Shot Transfer in Reinforcement Learning (2017, background): “Past attempts to build RL agents with strong domain adaptation performance highlighted the importance of learning good internal representations of raw observations (Finn et al., 2015; Raffin et al., 2017; Pan & Yang, 2009; Barreto et al., 2016; Littman et al., 2001).”
Visual Semantic Planning Using Deep Successor Representations (2017, methodology): “[3] implies a bound of the gap between value functions of the optimal policy π∗ n+1 and the policy π ′:”
Learning by Playing - Solving Sparse Reward Tasks from Scratch (2018, background): “These are in turn are inherently connected to learning to predict the future via “successor” representations (Dayan, 1993; Kulka-rni et al., 2016b; Barreto et al., 2017) or forecasts (Schaul & Ring, 2013; Lample & Chaplot, 2017; Dosovitskiy & Koltun, 2017) and are trained to be predictive of…”
Deep reinforcement learning with successor features for navigation across similar environments (2016, methodology): “This idea has recently been explored [10], [7].”
L2Explorer: A Lifelong Reinforcement Learning Assessment Environment (2022, background): “Research into measures of similarity, transfer, and generalization is ongoing (Barreto et al. 2016; Ma et al. 2020) and beyond the scope of this work.”
Contrastive Learning as Goal-Conditioned Reinforcement Learning (2022, background): “Prior work has studied how representations related to reward functions using the framework of universal value functions [11, 106] and successor features [8, 51, 81].”
Self-supervised Learning of Image Embedding for Continuous Control (2019, methodology): “This algorithm combines a Q ﬁtting done with Retrace [Munos et al., 2016] to propagate the discounted returns faster, and a policy improvement step.”
Generalized Hidden Parameter MDPs Transferable Model-based RL in a Handful of Trials (2020, background): “Learning across tasks with common dynamics has been approached with meta-reinforcement learning (Finn, Abbeel, and Levine 2017) or using successor features (Barreto et al. 2017).”