Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learning

By Abhishek Gupta, Coline Devin, Pieter Abbeel, Sergey Levine. 2017

292 citations recorded by the source corpus.

People can learn a wide range of tasks from their own experience, but can also learn from observing other creatures. This can accelerate acquisition of new skills even when the observed agent differs substantially from the learning agent in terms of morphology. In this paper, we examine how reinforcement learning algorithms can transfer knowledge between morphologically different agents (e.g., different robots). We introduce a problem formulation where two agents are tasked with learning multiple skills by sharing information. Our method uses the skills that were learned by both agents to train invariant feature spaces that can then be used to transfer other skills from one agent to another. The process of learning these invariant feature spaces can be viewed as a kind of "analogy making", or implicit learning of partial correspondences between two distinct domains. We evaluate our transfer learning algorithm in two simulated robotic manipulation skills, and illustrate that we can transfer knowledge between simulated robotic arms with different numbers of links, as well as simulated arms with different actuation mechanisms, where one robot is torque-driven while the other is tendon-driven.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Domain randomization for transferring deep neural networks from simulation to the real world (2017, background): “See [15] for a more complete treatment of domain adaptation in the reinforcement learning literature.”
One-Shot Imitation Learning (2017, background): “Success stories include domain adaptation in computer vision [64, 34, 28, 4, 15, 24, 33, 59, 14] and control [60, 45, 46, 20, 54].”
Solving Rubik's Cube with a Robot Hand (2019, background): “Domain adaption methods [112, 38], progressive nets [92], and learning inverse dynamics models [18] were all proposed to help with sim to real transfer.”
Driving Policy Transfer via Modularity and Abstraction (2018, background): “inly dealt with manual grasping and manipulation. A number of works employ specialized learning techniques and network architectures to facilitate transfer: for example, variants of domain adaptation [48,16,51,4,36] and specialized architectures [41,53]. Working with depth maps instead of color imag”
RoboNet: Large-Scale Multi-Robot Learning (2019, background): “A number of works have studied learning representations and policies that transfer across domains, including transfer from simulation to the real world [1, 33, 34], transfer across different dynamics [35, 36, 37, 2], transfer across robot morphologies with invariant feature spaces [38] and modularit”
Leveraging Human Guidance for Deep Reinforcement Learning Tasks (2019, methodology): “One method developed to approach this problem learns a correspondence between the embodiments using autoencoders in a supervised fashion [ Gupta et al. , 2018 ] .”
One-Shot Visual Imitation Learning via Meta-Learning (2017, background): “Sharing information across tasks is by no means a new idea, e.g. by using task-to-task mappings [22], gating [23], and shared features [24].”
Learning dexterous in-hand manipulation (2018, background): “Domain adaption methods [66, 21], progressive nets [53], and learning inverse dynamics models [10] were all proposed to help with sim to real transfer.”
DARLA: Improving Zero-Shot Transfer in Reinforcement Learning (2017, background): “…source and target domain representations by utilising observation and reward signals from both domains (Tzeng et al., 2016; Daftry et al., 2016; Parisotto et al., 2015; Guez et al., 2012; Talvitie & Singh, 2007; Niekum et al., 2013; Gupta et al., 2017; Finn et al., 2017; Rajendran et al., 2017).”
Sim-to-Real Transfer of Robotic Control with Dynamics Randomization (2017, background): “Tzeng et al. [11] and Gupta et al. [13] explored using pairwise constraints to encourage networks to learn similar embeddings for samples from different domains that are labeled as being similar.”
An Algorithmic Perspective on Imitation Learning (2018, neutral): “Recent work such as [Gupta et al., 2017, Finn et al., 2017a,b, Duan et al., 2017] addresses this research direction.”
Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task (2017, background): “ry the simulated control policy to decide on suitable actions to take in the real world. Transfer learning is concerned with transferring knowledge between different tasks or scenarios. Works such as [28,29] show skill transfer within simulation, whereas we concentrate on simulationto-real transfer.”
Behavioral Cloning from Observation (2018, background): “We assume that some of the state features are speciﬁcally related to the task and others speciﬁcally to the agent, i.e., a given state s can be partitioned into an agent-speciﬁc state , s a , and a task-speciﬁc state , s t , which are members of sets S a and S t , respectively (i.e., S = S a × S t )”
Benchmark Environments for Multitask Learning in Continuous Domains (2017, methodology): “…the modiﬁed environments; adding motor noise to the same set of environments (Christiano et al., 2016); simulated grasping and stacking using a Jaco arm (Rusu et al., 2016); and several custom grasping and manipulation tasks to demonstrate learning invariant feature spaces (Gupta et al., 2017).”
Recent Advances in Imitation Learning from Observation (2019, methodology): “One IfO method developed to address this problem learns a correspondence between the embodiments using autoencoders in a supervised fashion [Gupta et al., 2018].”
Exploiting Hierarchy for Learning and Transfer in KL-regularized RL (2019, background): “Works that transfer knowledge across different bodies include (Devin et al., 2017; Gupta et al., 2017; Sermanet et al., 2017; Xie et al., 2018).”
Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translation (2017, methodology): “The model (Fig 2), assumes that the demonstrations Di and Dj are aligned in time, though this assumption could be relaxed in future work by using iterative time alignment [35].”
Learning Transferable Knowledge Through Embedding Spaces (2019, background): “[42] address cross-agent transfer for two agents with deep models that learn multiple skills to handle RL tasks.”
Mutual Alignment Transfer Learning (2017, background): “[19] learn invariant feature spaces which are employed to transfer task knowledge even between systematically different robot architectures.”