Reinforcement Learning for Mapping Instructions to Actions

By S. Branavan, Harr Chen, Luke S. Zettlemoyer, Regina Barzilay. 2009

333 citations recorded by the source corpus.

In this paper, we present a reinforcement learning approach for mapping natural language instructions to sequences of executable actions. We assume access to a reward function that defines the quality of the executed actions. During training, the learner repeatedly constructs action sequences for a set of documents, executes those actions, and observes the resulting reward. We use a policy gradient algorithm to estimate the parameters of a log-linear model for action selection. We apply our method to interpret instructions in two domains --- Windows troubleshooting guides and game tutorials. Our results demonstrate that this method can rival supervised learning techniques while requiring few or no annotated training examples.

Subfields: Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Learning with Latent Language (2017, neutral): “…to a bilinear function of the encoded description and world state. f is effectively an instruction following model of a kind well-studied in the natural language processing literature (Branavan et al., 2009); the proposal model allows it to generate its own instructions without external direction.”
Joint Concept Learning and Semantic Parsing from Natural Language Explanations (2017, neutral): “Similar to our work, previous approaches have used different kinds of external-world signals to guide semantic interpretation (Liang et al., 2009; Branavan et al., 2009).”
Parsing Algebraic Word Problems into Equations (2015, background): “More closely related is work on language grounding, whose goal is the interpretation of a sentence in the context of a world representation (Branavan et al., 2009; Liang et al., 2009; Chen et al., 2010; Bordes et al., 2010; Feng and Lapata, 2010; Hajishirzi et al., 2011; Matuszek et al., 2012;…”
Learning to Solve Arithmetic Word Problems with Verb Categorization (2014, background): “Our work is also closely related to the grounded language acquisition research (Snyder and Barzi-lay, 2007; Branavan et al., 2009; Branavan et al., 2012; Vogel and Jurafsky, 2010; Chen et al., 2010; Hajishirzi et al., 2011; Chambers and Jurafsky, 2009; Liang et al., 2009; Bordes et al., 2010) where…”
Learning to Automatically Solve Algebra Word Problems (2014, background): “Techniques that consider multiple sentences typically do so in a serial fashion, processing each in turn with limited cross-sentence reasoning (Branavan et al., 2009; Zettlemoyer and Collins, 2009; Chen and Mooney, 2011; Artzi and Zettlemoyer, 2013).”
Learning Dependency-Based Compositional Semantics (2011, methodology): “For example, previous work learned to interpret language to troubleshoot aWindows machine (Branavan et al. 2009; Branavan, Zettlemoyer, and Barzilay 2010), win a game of Civilization (Branavan, Silver, and Barzilay 2011), play a legal game of solitaire (Eisenstein et al.”
Weakly Supervised Learning of Semantic Parsers for Mapping Instructions to Actions (2013, methodology): “This style of algorithm, especially when using the weaker end state validation, is closely related to reinforcement learning approaches (Branavan et al., 2009, 2010).”
Latent Predictor Networks for Code Generation (2016, background): “Finally, mapping natural language into a sequence of actions for the generation of executable code (Branavan et al., 2009).”
Learning to Interpret Natural Language Navigation Instructions from Observations (2011, background): “Some recent work has explored how to map natural-language instructions into actions that can be performed by a computer (Branavan et al. 2009; Lau, Drews, and Nichols 2009).”
Zero-shot Text Classification With Generative Language Models (2019, background): “…translation in low resource languages (Gu et al., 2018), iteratively guiding policies with language (Co-Reyes et al., 2018) for instruction following (Branavan et al., 2009; Chen and Mooney, 2011), and generating WikiSQL-style structured queries from natural language queries (Huang et al., 2018).”
Speaker-Follower Models for Vision-and-Language Navigation (2018, background): “Systems that learn to carry out natural language instructions in an interactive environment include approaches based on intermediate structured and executable representations of language [50, 9, 4, 28, 20] and approaches that map directly from language and world state observations to actions [7, 2, ”
Learning Language Games through Interaction (2016, background): “lated Work and Discussion Our work connects with a broad body of work on grounded language, in which language is used in some environment as a means towards some goal. Examples include playing games (Branavan et al., 2009, 2010; Reckman et al., 2010) interacting with robotics (Tellex et al., 2011, 2”
Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning (2017, background): “Although Branavan et al. (2009) also tackle a similar problem, the agent is given a single instruction at a time, while our agent needs to learn how to align instructions and state given a full list of instructions.”
Learning to Win by Reading Manuals in a Monte-Carlo Framework (2011, methodology): “Within this line of work, we are most closely related to reinforcement learning approaches that learn language by proactively interacting with an external environment (Branavan et al., 2009; Branavan et al., 2010; Vogel and Jurafsky, 2010).”
Mapping Instructions and Visual Observations to Actions with Reinforcement Learning (2017, background): “…and Mooney, 2011; Kim and Mooney, 2012, 2013; Artzi and Zettlemoyer, 2013; Artzi et al., 2014a,b; Misra et al., 2015, 2016), alignment models (Andreas and Klein, 2015), reinforcement learning (Branavan et al., 2009, 2010; Vogel and Jurafsky, 2010), and neural network models (Mei et al., 2016).”
Deep Reinforcement Learning with a Natural Language Action Space (2015, background): “, 2011), and mapping text instructions to sequences of executable actions (Branavan et al., 2009).”
Reading to Learn: Constructing Features from Semantic Abstracts (2009, methodology): “…on extracting logical forms from surface text (Zettlemoyer and Collins, 2005; Ge and Mooney, 2005; Downey et al., 2005; Liang et al., 2009), interpreting language in the context of a domain (Chen and Mooney, 2008), and using an actionable domain to guide text interpretation (Branavan et al., 2009).”