Language as an Abstraction for Hierarchical Deep Reinforcement Learning

By Yiding Jiang, Shixiang Shane Gu, Kevin P. Murphy, Chelsea Finn. 2019

248 citations recorded by the source corpus.

Solving complex, temporally-extended tasks is a long-standing problem in reinforcement learning (RL). We hypothesize that one critical element of solving such problems is the notion of compositionality. With the ability to learn concepts and sub-skills that can be composed to solve longer tasks, i.e. hierarchical RL, we can acquire temporally-extended behaviors. However, acquiring effective yet general abstractions for hierarchical RL is remarkably challenging. In this paper, we propose to use language as the abstraction, as it provides unique compositional structure, enabling fast learning and combinatorial generalization, while retaining tremendous flexibility, making it suitable for a variety of problems. Our approach learns an instruction-following low-level policy and a high-level policy that can reuse abstractions across tasks, in essence, permitting agents to reason using structured language. To study compositional task learning, we introduce an open-source object interaction environment built using the MuJoCo physics engine and the CLEVR engine. We find that, using our approach, agents can learn to solve to diverse, temporally-extended tasks such as object sorting and multi-object rearrangement, including from raw pixel observations. Our analysis reveals that the compositional nature of language is critical for learning diverse sub-skills and systematically generalizing to new sub-skills in comparison to non-compositional abstractions that use the same supervision.

Subfields: Artificial Intelligence, Mechanical Engineering, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Mechanical Engineering

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, background): “Most of these prior works focus on following low-level instructions, such as for pick-and-place tasks and other robotic manipulation primitives [20, 22, 56, 21, 13, 26], though some methods address long-horizon, compound tasks in simulated domains [57, 58, 54].”
Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, background): “Others have approached the problem through hierarchical learning [45, 46, 34, 47, 48, 49, 50].”
Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “, 2011); and other areas such as hierarchical RL (Peng et al., 2017; Co-Reyes et al., 2018; Jiang et al., 2019).”
Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text (2020, methodology): “Much recent work applies deep learning and policy optimization in end-to-end approaches to learning instruction following [Chaplot et al., 2018, Oh et al., 2017, Bahdanau et al., 2018, Chevalier-Boisvert et al., 2018, Yu et al., 2018b,a, Jiang et al., 2019].”
Language Conditioned Imitation Learning Over Unstructured Data (2020, background): “binary pick and place primitives [18, 48]), or synthetic predefined language [17, 6, 20, 54].”
Can Wikipedia Help Offline Reinforcement Learning? (2022, background): “…al., 2020; Tirumala et al., 2020), due to the difﬁculty in parameterizing different scenes and tasks through a single network (Wang et al., 2018b; Jiang et al., 2019; Zeng et al., 2020) as well as the lack of large off-the-shelf datasets for pre-training (Cobbe et al., 2020; Zhu et al., 2020; Yu…”
RTFM: Generalising to Novel Environment Dynamics via Reading (2019, background): “In addition to jointly learning policies based on external documentation and language goals, we are interested in exploring how to use supporting evidence in external documentation to reason about plans (Andreas et al., 2018) and induce hierarchical policies (Hu et al., 2019; Jiang et al., 2019).”
Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks (2021, background): “One class of prior work is hierarchical RL, which aims to learn temporally-extended primitive skills and a high-level policy over them [4, 5, 6, 7, 8, 58].”
Tell me why! Explanations support learning relational and causal structure (2021, methodology): “Language in RL: Language is used broadly in RL, whether as instructions (e.g. Hermann et al., 2017; Kaplan et al., 2017), to target exploration (Goyal et al., 2019; Watkins et al., 2021; Mu et al., 2022; Tam et al., 2022), or as an abstraction to structure hierarchical policies (Jiang et al., 2019).”
One-Shot Learning from a Demonstration with Hierarchical Latent Language (2022, unknown): “2 Jiang et al. (2019) train hierarchical synthetic language policy agents to accomplish a set of shape-arranging tasks in a MuJoCo-based environment.”
Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation (2021, methodology): “We also compare to language-conditioned RL (LCRL), which labels the final state in each episode as having reward 1 for the annotated language instruction and 0 elsewhere, and trains a language conditioned-policy using offline Q learning, which reflects a fully offline version of the low-level policy”
A Metaverse: Taxonomy, Components, Applications, and Open Challenges (2022, unknown): “Jiang et al. [158] used a language that is ﬂexibly applied to the generalization of various goals, rapid training, and combinations as an abstraction to solve the difﬁculty of generalized abstraction in hierarchical RL. AQM (Questioner’s Mind) agents ask more consistent questions to maximize…”
Code as Policies: Language Model Programs for Embodied Control (2022, unknown): “, model-based [33]–[35], imitation learning [8], [9], [36], [37], or reinforcement learning [38]–[42]).”
ProgPrompt: Generating Situated Robot Task Plans using Large Language Models (2022, background): “Recent works have explored learning-based task & motion planning, using methods such as representation learning, hierarchical learning, language as planning space, learning compositional skills and more [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26].”
Vygotskian Autotelic Artificial Intelligence: Language and Culture Internalization for Human-Like AI (2022, background): “Linguistic autotelic agents can reach and make sense of abstract goals like “ go to any circle ,” 87 relational goals “ sort objects by size ,” 88 “ put the cylinder in the drawer ,” 89 sequential goals “ open the yellow door after opening a red door ,” 90 or even learning goals “ is the ghargh…”