Modular Multitask Reinforcement Learning with Policy Sketches

By Jacob Andreas, Daniel Klein, Sergey Levine. 2016

510 citations recorded by the source corpus.

We describe a framework for multitask deep reinforcement learning guided by policy sketches. Sketches annotate tasks with sequences of named subtasks, providing information about high-level structural relationships among tasks but not how to implement them—specifically not providing the detailed guidance used by much previous work on learning policy abstractions for RL (e.g. intermediate rewards, subtask completion signals, or intrinsic motivations). To learn from sketches, we present a model that associates every subtask with a modular subpolicy, and jointly maximizes reward over full task-specific policies by tying parameters across shared subpolicies. Optimization is accomplished via a decoupled actor-critic training objective that facilitates learning common behaviors from multiple dissimilar reward functions. We evaluate the effectiveness of our approach in three environments featuring both discrete and continuous control, and with sparse rewards that can be obtained only after completing a number of high-level sub-goals. Experiments show that using our approach to learn policies guided by sketches gives better performance than existing techniques for learning task-specific or shared policies, while naturally inducing a library of interpretable primitive behaviors that can be recombined to rapidly adapt to new tasks.

Subfields: Artificial Intelligence, Control and Systems Engineering, Information Systems, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Information Systems

In Reinforcement Learning

Citations in the corpus

Environmental drivers of systematicity and generalization in a situated agent (2019, background): “Indeed, prior work has found that language can serve to promote compositional behaviour in deep RL agents (Andreas et al., 2017).”
Learning with Latent Language (2017, neutral): “Structured language-like annotations have been used to improve learning of generalizable structured policies (Oh et al., 2017; Andreas et al., 2017; Denil et al., 2017).”
Training Language Models with Language Feedback (2022, background): “, 2017, inter alia), and train the model via strong supervision (Andreas et al., 2017; Kaplan et al., 2017), reward shaping (Goyal et al.”
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, background): “Most of these prior works focus on following low-level instructions, such as for pick-and-place tasks and other robotic manipulation primitives [20, 22, 56, 21, 13, 26], though some methods address long-horizon, compound tasks in simulated domains [57, 58, 54].”
Grammar-Based Grounded Lexicon Learning (2022, background): “Integrating symbolic structures such as programs and neural networks has shown success in modeling compositional queries in various domains, including image and video reasoning [20, 31], knowledge base query [4], and robotic planning [6].”
Neural Logic Machines (2019, background): “Besides, modular networks (Andreas et al., 2016; 2017; Mascharka et al., 2018) are proposed for the reasoning over subsymbolic data such as images and natural language question answering.”
MAttNet: Modular Attention Network for Referring Expression Comprehension (2018, methodology): “Since their introduction, modular networks have been applied to several other tasks: visual reasoning [7, 11], question answering [2], relationship modeling [9], multitask reinforcement learning [1], etc.”
Embodied Question Answering (2017, background): “Hierarchical modeling has recently shown promise in deep reinforcement learning [22,28,36].”
A Survey on Multi-Task Learning (2017, background): “In [138], a model associated each subtask with a modular subpolicy is proposed to learn from policy sketches, which annotate tasks with sequences of named subtasks and provide information about high-level structural relationships among tasks.”
Relational inductive biases, deep learning, and graph networks (2018, background): “…Andreas et al., 2016), partial tree traversals in a state-action graph (Guez et al., 2018; Farquhar et al., 2018), hierarchical action policies (Andreas et al., 2017), multi-agent communication channels (Foerster et al., 2016), “capsules” (Sabour et al., 2017), and programs (Parisotto et al.,…”
Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning (2017, background): “Besides visual grounding, Andreas et al. (2017) grounded instructions (not necessarily using human language) to local policies in hierarchical reinforcement learning.”
Disentangling Transfer in Continual Reinforcement Learning (2022, background): “Other lines of work exploit the model structure [36, 13] or enforce modularity [3, 9].”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Andreas, Klein, and Levine (2017); Oh, Singh, Lee, and Kohli (2017) assume known information about high-level structural relationships among tasks and similarity between different subtasks, respectively, to enable fast learning across tasks.”
From Few to More: Large-scale Dynamic Multiagent Curriculum Learning (2019, methodology): “Andreas et al. (2017) used curriculum learning to make their model scale up smoothly from simple tasks to difﬁcult ones, the difﬁculty of each task is associated with sketches of different length.”
Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning (2017, background): “Andreas et al. (2016) proposed a framework to learn the underlying subtasks from a policy sketch which speci-ﬁes a sequence of subtasks, and the agent can generalize over new sequences of them in principle.”
Building Generalizable Agents with a Realistic and Rich 3D Environment (2018, methodology): “More recently, end-to-end learning methods were introduced to predict robotic actions from raw pixel data (Levine et al., 2016).”
A Survey of Reinforcement Learning Informed by Natural Language (2019, methodology): “When the vocabulary of instructions is sufﬁciently simple, an explicit options policy can be constructed that associates each task description with its own modular sub-policy [ Andreas et al. , 2017].”
Measuring abstract reasoning in neural networks (2018, background): “This ﬁnding aligns with previous observations about the beneﬁts of discrete channels for knowledge representation (Andreas et al., 2016) and the beneﬁt of inducing explanations or rationales (Ling et al., 2017).”
Skill Induction and Planning with Latent Language (2021, methodology): “Our approach is closely related to previous work on learning skill libraries from policy sketches (Andreas et al., 2017; Shiarlis et al., 2018); instead of the fixed skill inventory used by policy sketches, (SL)3 learns an open-ended, compositional library of behaviors indexed by natural language st”