Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvement

By André Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel Jaymin Mankowitz, Augustin Žídek, R´emi Munos. 2018

192 citations recorded by the source corpus.

The ability to transfer skills across tasks has the potential to scale up reinforcement learning (RL) agents to environments currently out of reach. Recently, a framework based on two ideas, successor features (SFs) and generalised policy improvement (GPI), has been introduced as a principled way of transferring skills. In this paper we extend the SFs & GPI framework in two ways. One of the basic assumptions underlying the original formulation of SFs & GPI is that rewards for all tasks of interest can be computed as linear combinations of a fixed set of features. We relax this constraint and show that the theoretical guarantees supporting the framework can be extended to any set of tasks that only differ in the reward function. Our second contribution is to show that one can use the reward functions themselves as features for future tasks, without any loss of expressiveness, thus removing the need to specify a set of features beforehand. This makes it possible to combine SFs & GPI with deep learning in a more stable way. We empirically verify this claim on a complex 3D environment where observations are images from a first-person perspective. We show that the transfer promoted by SFs & GPI leads to very good policies on unseen tasks almost instantaneously. We also describe how to learn policies specialised to the new tasks in a way that allows them to be added to the agent's set of skills, and thus be reused in the future.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

Citations in the corpus

RUDDER: Return Decomposition for Delayed Rewards (2018, background): “Successor features decouple environment and policy from rewards, but changing the reward changes the optimal policies [7, 6].”
How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition (2022, background): “A similar result showed that successor features can be combined to solve this type of combined objectives (Barreto et al., 2018).”
γ-Models: Generative Temporal Difference Learning for Infinite-Horizon Prediction (2020, background): “Prior continuous variants have focused on policy evaluation based on expected state featurizations (Barreto et al., 2017, 2018; Hansen et al., 2020), forgoing an interpretation as a probabilistic model suitable for state prediction.”
Learning Structures: Predictive Representations, Replay, and Generalization (2020, methodology): “The RL framework can be used to learn mappings between observed perceptual features and states [25], compact representations of relational and associative structures of the state space [15 ,23,26 ], as well as abstractions enabling transfer between tasks and environments [27,28].”
Self-supervised Learning of Image Embedding for Continuous Control (2019, background): “Successor features [Dayan, 1993] or representations [Barreto et al., 2018, 2016] constitute another related body of work.”
Modular Lifelong Reinforcement Learning via Neural Composition (2022, background): “One additional form of composition in RL uses full task policies as the components, and combines policies together when solving a new task (e.g., by summing their value functions; Todorov, 2009; Barreto et al., 2018; Haarnoja et al., 2018; Van Niekerk et al., 2019; Nangue Tasse et al., 2020).”
The Option Keyboard: Combining Skills in Reinforcement Learning (2021, methodology): “Previous work has used GPI and successor features, the linear form of GPE considered here, in the context of transfer [3, 4, 5].”