Mutual Alignment Transfer Learning

By Markus Wulfmeier, Ingmar Posner, Pieter Abbeel. 2017

63 citations recorded by the source corpus.

Training robots for operation in the real world is a complex, time consuming and potentially expensive task. Despite significant success of reinforcement learning in games and simulations, research in real robot applications has not been able to match similar progress. While sample complexity can be reduced by training policies in simulation, such policies can perform sub-optimally on the real platform given imperfect calibration of model dynamics. We present an approach -- supplemental to fine tuning on the real robot -- to further benefit from parallel access to a simulator during training and reduce sample requirements on the real robot. The developed approach harnesses auxiliary rewards to guide the exploration for the real world agent based on the proficiency of the agent in simulation and vice versa. In this context, we demonstrate empirically that the reciprocal alignment for both agents provides further benefit as the agent in simulation can adjust to optimize its behaviour for states commonly visited by the real-world agent.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

Citations in the corpus

Driving Policy Transfer via Modularity and Abstraction (2018, background): “inly dealt with manual grasping and manipulation. A number of works employ specialized learning techniques and network architectures to facilitate transfer: for example, variants of domain adaptation [48,16,51,4,36] and specialized architectures [41,53]. Working with depth maps instead of color imag”
Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience (2018, background): “stribution of simulated parameters, but also learn a component that predicts the system parameters from the current states and actions, and use the prediction as an additional input to the policy. In [22], an auxiliary reward is used to encourage policies trained in source and target environments to”
Sim-to-Real Transfer of Robotic Control with Dynamics Randomization (2017, background): “In the context of RL, adversarial losses have been used to transfer policies between different simulated domains, by encouraging agents to adopt similar behaviours across the various environments [15].”
Regularized Hierarchical Policies for Compositional Transfer in Robotics (2019, background): “…Finn et al., 2017], sharing models and parameters across tasks [e.g. Rusu et al., 2016, Teh et al., 2017, Galashov et al., 2018], data-sharing across tasks [e.g. Riedmiller et al., 2018, Andrychowicz et al., 2017] or related, auxiliary objectives [Jaderberg et al., 2016, Wulfmeier et al., 2017].”
DDCO: Discovery of Deep Continuous Options for Robot Learning from Demonstrations (2017, background): “There are also several approaches that consider “demonstration clustering”, that is, identifying clusters of already segmented demonstrations [25, 26, 27, 28].”
NerveNet: Learning Structured Policy with Graph Neural Networks (2018, methodology): “In Wulfmeier et al. (2017), the authors propose a method of transfer learning by using imitation learning.”
Neural Logic Reinforcement Learning (2019, methodology): “However, similar to traditional reinforcement learning algorithms such as tabular TD-learning (Sutton & Barto, 1998), DRL algorithms can only learn policies that are hard to interpret (Montavon et al.) and cannot be generalized from one environment to another similar one (Wulfmeier et al., 2017).”