Hierarchical Decision Making by Generating and Following Natural Language Instructions

By Hengyuan Hu, Denis Yarats, Qucheng Gong, Yuandong Tian, Michael Lewis. 2019

70 citations recorded by the source corpus.

We explore using latent natural language instructions as an expressive and compositional representation of complex actions for hierarchical decision making. Rather than directly selecting micro-actions, our agent first generates a latent plan in natural language, which is then executed by a separate model. We introduce a challenging real-time strategy game environment in which the actions of a large number of units must be coordinated across long time scales. We gather a dataset of 76 thousand pairs of instructions and executions from human play, and train instructor and executor models. Experiments show that models using natural language as a latent variable significantly outperform models that directly imitate human actions. The compositional structure of language proves crucial to its effectiveness for action representation. We also release our code, models and data.

Subfields: Artificial Intelligence, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

A Survey of Reinforcement Learning Informed by Natural Language (2019, background): “More recently, [Hu et al. , 2019] consider generated natural language as a representation for macro actions in a real-time strategy game environment based on [Tian et al. , 2017].”
Skill Induction and Planning with Latent Language (2021, background): “As a result, all existing work on language-based policy learning has focused on very short time horizons (Andreas et al., 2018), restricted language (Hu et al., 2019; Jacob et al., 2021) or synthetic training data (Shu et al., 2018; Jiang et al., 2019).”
Boosting Search Engines with Interactive Agents (2021, neutral): “Most of the work in language-related RL is otherwise centered on synthetic navigation/arcade environments (Hu et al., 2019).”
Reason first, then respond: Modular Generation for Knowledge-infused Dialogue (2021, methodology): “In reinforcement learning settings, generating natural language has been used as an intermediate planning step (Sharma et al., 2021; Hu et al., 2019), and in particular in goal-oriented dialogue (Yarats and Lewis, 2018) and open-domain QA (Adolphs et al., 2021) as well.”
Multitasking Inhibits Semantic Drift (2021, methodology): “LLPs have been used to construct interactive agents capable of complex reasoning (e.g. programming by demonstration) and planning over long horizons (e.g. in strategy games; Hu et al., 2019).”
Vygotskian Autotelic Artificial Intelligence: Language and Culture Internalization for Human-Like AI (2022, background): “88,105,106,108 Here again, agents internalize linguistic production to autonomously generate further guidance to themselves (task decompositions).”
Learning to Annotate: Modularizing Data Augmentation for Text Classifiers with Natural Language Explanations (2019, methodology): “Hu et al. (2019) incorporate human knowledge into neural networks by using a teacher network to teach the classifier knowledge from rules and train the classifier with labeled data.”
Improving Intrinsic Exploration with Language Abstractions (2022, background): “, 2021), or use language for hierarchical planning (Andreas et al., 2018; Jiang et al., 2019; Hu et al., 2019; Chen et al., 2021; Prakash et al., 2021).”
Executing Instructions in Situated Collaborative Interactions (2019, unknown): “…only one side acts in the world (Anderson et al., 1991; Williams et al., 2013; Vlachos and Clark, 2014; de Vries et al., 2018; Kim et al., 2019; Hu et al., 2019), coordination for agreed selection of an object (He et al., 2017; Udagawa and Aizawa, 2019), and negotiation (Lewis et al., 2017; He…”
One-Shot Learning from a Demonstration with Hierarchical Latent Language (2022, unknown): “Instructor module Our framework for generating and following instructions given a task description is similar to that of Hu et al. (2019), except we use a language model decoder instead of a classiﬁer and compute separate state encodings for the two modules.”
Language is Power: Representing States Using Natural Language in Reinforcement Learning. (2019, methodology): “In a paper by Hu et al. [2019], instructions given in natural language were used in order to break down complex problems into high-level plans and lower-level actions.”