Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning

By Tianmin Shu, Caiming Xiong, Richard Socher. 2017

150 citations recorded by the source corpus.

Learning policies for complex tasks that require multiple different skills is a major challenge in reinforcement learning (RL). It is also a requirement for its deployment in real-world scenarios. This paper proposes a novel framework for efficient multi-task reinforcement learning. Our framework trains agents to employ hierarchical policies that decide when to use a previously learned policy and when to learn a new skill. This enables agents to continually acquire new skills during different stages of training. Each learned task corresponds to a human language description. Because agents can only access previously learned skills through these descriptions, the agent can always provide a human-interpretable description of its choices. In order to help the agent learn the complex temporal dependencies necessary for the hierarchical policy, we provide it with a stochastic temporal grammar that modulates when to rely on previously learned skills and when to execute new skills. We validate our approach on Minecraft games designed to explicitly test the ability to reuse previously learned skills while simultaneously learning new skills.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

In Meta & Continual Learning

Citations in the corpus

Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, background): “A second set of assumptions use hierarchical policies (Shu et al., 2017) that are learned over multiple tasks, where each task is decomposed into multiple sub-tasks (skills).”
A Survey on Multi-Task Learning (2017, background): “For example, in [162], a MRL framework is proposed to”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Meanwhile, Shu, Xiong, and Socher (2017); Geishauser, van Niekerk, Lubis, Heck, Lin, Feng, and Gašić (2022) employ weak supervision from humans to define what skills should be learned in the form of language instructions.”
A Survey of Reinforcement Learning Informed by Natural Language (2019, background): “…language and its compositionality make it a particularly good candidate for representing policies in hierarchical RL. [Andreas et al. , 2017] and [Shu et al. , 2018] can be viewed as using language (rather than logical or learned representations) as policy speciﬁca-tions for a hierarchical agent.”
Language as an Abstraction for Hierarchical Deep Reinforcement Learning (2019, methodology): “While prior work has made use of language-based sub-goal policies in hierarchical RL [51, 13], the instruction representation used lack the needed diversity to benefit from the compositionality of language over one-hot goal representations.”
Learning Abstract Options (2018, background): “In [37], the authors propose to ground all goals in a natural language description space.”
MineRL: A Large-Scale Dataset of Minecraft Demonstrations (2019, methodology): “As a result of the large number of task-level annotations, the dataset can be used for large-scale option extraction and skill acquisition, enabling the extension of the work of [Shu et al. , 2017] and [Andreas et al. , 2017].”
MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale (2021, background): “Learning complex and composite skills has been addressed through hierarchical reinforcement learning with options [75, 7, 14], combining multiple sub-tasks [6, 16, 49, 26, 69, 89], reusing samples between tasks [39], relabeling experience in hindsight [3], introducing demonstrations [58, 81, 29, 41,”
Multi-Task Reinforcement Learning with Context-based Representations (2021, methodology): “For example, (Shu et al., 2017) uses “a two-word tuple template consisting of a skill and an item” to describe the task while the metadata for CARE can be high-level, under-specified, and unstructured.”
Unity: A General Platform for Intelligent Agents (2018, methodology): “As a result, there have been a number of research projects exploring multi-agent communication, hierarchical control, and planning using the platform (Oh et al., 2016; Shu et al., 2017; Tessler et al., 2017).”
Why Build an Assistant in Minecraft? (2019, background): “Some of these use (templated) language to describe mid-level macros [71] or tasks [67].”
Learning to Synthesize Programs as Interpretable and Generalizable Policies (2021, unknown): “[84] Richard Socher Tianmin Shu, Caiming Xiong.”
Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos (2022, background): “[57] Tianmin Shu, Caiming Xiong, and Richard Socher.”
VRKitchen: an Interactive 3D Virtual Environment for Task-oriented Learning (2019, background): “Recently, many researchers have focused on learning hierarchical policies [3, 44, 47] in simple domains.”
Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration (2020, methodology): “This is inspired by a common goal-conditioned policy network architecture (Chaplot et al., 2018; Shu et al., 2017), which helps extract state information relevant to the goal.”
Multitasking Inhibits Semantic Drift (2021, background): “Building on this intuition, several recent papers have explored hierarchical RL in which natural language is used to parameterize the space of high-level actions (Oh et al., 2017; Andreas et al., 2017; Shu et al., 2018; Jiang et al., 2019; Hu et al., 2019).”