Regularized Hierarchical Policies for Compositional Transfer in Robotics

By Markus Wulfmeier, Abbas Abdolmaleki, Roland Hafner, Jost Tobias Springenberg, Michael Neunert, Tim Hertweck, Thomas Lampe, Noah Siegel, Nicolas Manfred Otto Heess, Martin A. Riedmiller. 2019

27 citations recorded by the source corpus.

The successful application of flexible, general learning algorithms -- such as deep reinforcement learning -- to real-world robotics applications is often limited by their poor data-efficiency. Domains with more than a single dominant task of interest encourage algorithms that share partial solutions across tasks to limit the required experiment time. We develop and investigate simple hierarchical inductive biases -- in the form of structured policies -- as a mechanism for knowledge transfer across tasks in reinforcement learning (RL). To leverage the power of these structured policies we design an RL algorithm that enables stable and fast learning. We demonstrate the success of our method both in simulated robot environments (using locomotion and manipulation domains) as well as real robot experiments, demonstrating substantially better data-efficiency than competitive baselines.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

A Framework for Data-Driven Robotics (2019, neutral): “[73] also attempted the block stacking task.”
Gradient Surgery for Multi-Task Learning (2020, methodology): “The multi-task formulation has been applied to many different settings, including supervised learning [63, 35, 60, 53, 62] and reinforcement-learning [17, 58], as well as many different domains, such as vision [3, 39, 31, 33, 62], language [11, 15, 38, 44] and robotics [45, 59, 25].”
Scaling data-driven robotics with reward sketching and batch reinforcement learning (2019, methodology): “[72] also attempted reward learning with the block stacking task.”
A Distributional View on Multi-Objective Policy Optimization (2020, background): “Generally these algorithms learn a separate policy for each task, with shared learning across tasks (Teh et al., 2017; Riedmiller et al., 2018; Wulfmeier et al., 2019).”
Sharing Knowledge in Multi-Task Deep Reinforcement Learning (2020, background): “While some of these works rely on a single deep neural network to model the multi-task agent (Liu et al., 2016; Yang et al., 2017; Hessel et al., 2018; Wulfmeier et al., 2019), others use multiple deep neural networks, e.”