Learning modular neural network policies for multi-task and multi-robot transfer

By Coline Devin, Abhishek Gupta, Trevor Darrell, Pieter Abbeel, Sergey Levine. 2016

453 citations recorded by the source corpus.

Reinforcement learning (RL) can automate a wide variety of robotic skills, but learning each new skill requires considerable real-world data collection and manual representation engineering to design policy classes or features. Using deep reinforcement learning to train general purpose neural network policies alleviates some of the burden of manual representation engineering by using expressive policy classes, but exacerbates the challenge of data collection, since such methods tend to be less efficient than RL with low-dimensional, hand-designed representations. Transfer learning can mitigate this problem by enabling us to transfer information from one skill to another and even from one robot to another. We show that neural network policies can be decomposed into “task-specific” and “robot-specific” modules, where the task-specific modules are shared across robots, and the robot-specific modules are shared across all tasks on that robot. This allows for sharing task information, such as perception, between robots and sharing robot information, such as dynamics and kinematics, between tasks. We exploit this decomposition to train mix-and-match modules that can solve new robot-task combinations that were not seen during training. Using a novel approach to train modular neural networks, we demonstrate the effectiveness of our transfer method for enabling zero-shot generalization with a variety of robots and tasks in simulation for both visual and non-visual tasks.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

In Model Architectures

Citations in the corpus

RL$^2$: Fast Reinforcement Learning via Slow Reinforcement Learning (2016, background): “There has been a growing interest in using deep neural networks for multi-task learning and transfer learning (Parisotto et al., 2015; Rusu et al., 2015; 2016a; Devin et al., 2016; Rusu et al., 2016b).”
Programmable Agents (2017, background): “Compositionality has become a topic of great interest in machine learning, robotics and cognitive science [20, 30, 19, 9, 35, 26], and it has been central to research in language [32, 25, 33, 36].”
Disentangling Transfer in Continual Reinforcement Learning (2022, background): “Other lines of work exploit the model structure [36, 13] or enforce modularity [3, 9].”
Model Zoo: A Growing Brain That Learns Continually (2021, methodology): “…has been recognized that this is not suﬃcient; this has given rise to methods for soft-parameter sharing that either design or learn specialized routing architectures (Rosenbaum et al., 2017; Sun et al., 2019; Fernando et al., 2017; Devin et al., 2017; Misra et al., 2016; Vandenhende et al., 2019).”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “More recent work (Devin et al., 2017; Frans et al., 2017; Fernando et al., 2017; Rosenbaum et al., 2017; Meyerson and Miikkulainen, 2017; Ramachandran and Le, 2018; Chang et al., 2018; Liang et al., 2018; Alet et al., 2018; Cases et al., 2019; Yang et al., 2020) has focused on training neural networ”
Driving Policy Transfer via Modularity and Abstraction (2018, background): “res special instrumentation (AR tags) and was developed in the relatively constrained context of detecting and pushing an object with a robot arm against a uniform green-screen backdrop. Devin et al. [10] also investigate the beneﬁts of modularity in neural networks for control, but only evaluate in”
RoboNet: Large-Scale Multi-Robot Learning (2019, background): “A number of works have studied learning representations and policies that transfer across domains, including transfer from simulation to the real world [1, 33, 34], transfer across different dynamics [35, 36, 37, 2], transfer across robot morphologies with invariant feature spaces [38] and modularit”
RLBench: The Robot Learning Benchmark & Learning Environment (2019, background): “2974707 there is now a large body of work looking at increasing the capabilities of robotic agents through the use of reinforcement learning [3], [4], meta-learning [5]–[7], multi-task learning [8],”
Hindsight Experience Replay (2017, background): “Learning simultaneously policies for multiple tasks have been heavily explored in the context of policy search, e.g. Schmidhuber and Huber (1990); Caruana (1998); Da Silva et al. (2012); Kober et al. (2012); Devin et al. (2016); Pinto and Gupta (2016).”
Modular Multitask Reinforcement Learning with Policy Sketches (2016, methodology): “Such models have been previously used for tasks involving question answering (Iyyer et al., 2014; Andreas et al., 2016) and relational reasoning (Socher et al., 2012), and more recently for multi-task, multi-robot transfer problems (Devin et al., 2016).”
Gradient Surgery for Multi-Task Learning (2020, background): “A number of architectural solutions have been proposed to the multi-task learning problem based on multiple modules or paths (Fernando et al., 2017; Devin et al., 2016; Misra et al., 2016b; Rusu et al., 2016b; Rosenbaum et al., 2018; Vandenhende et al., 2019; Rosenbaum et al., 2018), or using…”
Modular meta-learning (2018, background): “Devin et al. [18] use a ﬁxed composition of two functions, one related to the robot and one to the task.”
Automatically Composing Representation Transformations as a Means for Generalization (2018, background): “The second learns the transformations but pre-speciﬁes the structure [5, 21, 66].”
Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning (2017, background): “Devin et al. (2017) proposed composing sub-networks that are shared across tasks and robots in order to achieve generalization to unseen conﬁgurations of them.”
Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learning (2017, methodology): “Our own recent work has looked at morphological differences in the context of multi-agent and multi-task learning (Devin et al., 2016), by reusing neural network components across agent/task combinations.”
Stochastic Neural Networks for Hierarchical Reinforcement Learning (2016, background): “…studied in the past in sequential settings, where the agent is exposed to a sequence of tasks, and should learn to make use of experience gathered from solving earlier tasks to help solve later tasks (Taylor & Stone, 2009; Wilson et al., 2007; Lazaric & Ghavamzadeh, 2010; Devin et al., 2017).”
Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task (2017, background): “ry the simulated control policy to decide on suitable actions to take in the real world. Transfer learning is concerned with transferring knowledge between different tasks or scenarios. Works such as [28,29] show skill transfer within simulation, whereas we concentrate on simulationto-real transfer.”
A Generalist Agent (2022, background): “Devin et al. (2017) learn a modular policy for multi-task and multi-robot transfer in simulated 2D manipulation environments.”
Vision-Based Multi-Task Manipulation for Inexpensive Robots Using End-to-End Learning from Demonstration (2017, background): “Neural networks policies can be decomposed to different modules where there are some taskspecific modules and some robot-specific modules [25].”