VPE: Variational Policy Embedding for Transfer Reinforcement Learning

By Isac Arnekvist, Danica Kragic, Johannes Andreas Stork. 2018

40 citations recorded by the source corpus.

Reinforcement Learning methods are capable of solving complex problems, but resulting policies might perform poorly in environments that are even slightly different. In robotics especially, training and deployment conditions often vary and data collection is expensive, making retraining undesirable. Simulation training allows for feasible training times, but on the other hand suffer from a reality-gap when applied in real-world settings. This raises the need of efficient adaptation of policies acting in new environments.We consider the problem of transferring knowledge within a family of similar Markov decision processes. We assume that Q-functions are generated by some low-dimensional latent variable. Given such a Q-function, we can find a master policy that can adapt given different values of this latent variable. Our method learns both the generative mapping and an approximate posterior of the latent variables, enabling identification of policies for new tasks by searching only in the latent space, rather than the space of all policies. The low-dimensional space, and master policy found by our method enables policies to quickly adapt to new environments. We demonstrate the method on both a pendulum swing-up task in simulation, and for simulation-to-real transfer on a pushing task.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

From Few to More: Large-scale Dynamic Multiagent Curriculum Learning (2019, methodology): “This concept is widely used in domain adaptation area (Higgins et al. 2017; Arnekvist, Kragic, and Stork 2019), while they focus on how to transfer from different tasks with the same state dimension.”
VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning (2019, methodology): “Arnekvist et al. (2019) learn a stochastic embedding of optimal Q-functions for different skills, such that the policy can be conditioned on (samples of) this embedding. When learning a new task, adaptation is done in latent space. Co-Reyes et al. (2018) learn a latent space of low-level skills that”
Fast Adaptation to New Environments via Policy-Dynamics Value Functions (2020, background): “In both Hausman et al. (2018) and Arnekvist et al. (2018), adaptation to a new task is done in the latent space with no further updates to the policy network.”