Learning to Win by Reading Manuals in a Monte-Carlo Framework

By S. Branavan, David Silver, Regina Barzilay. 2011

195 citations recorded by the source corpus.

This paper presents a novel approach for leveraging automatically extracted textual knowledge to improve the performance of control applications such as games. Our ultimate goal is to enrich a stochastic player with high-level guidance expressed in text. Our model jointly learns to identify text that is relevant to a given game state in addition to learning game strategies guided by the selected text. Our method operates in the Monte-Carlo search framework, and learns both text analysis and game strategies based only on environment feedback. We apply our approach to the complex strategy game Civilization II using the official game manual as the text guide. Our results show that a linguistically-informed game-playing agent significantly outperforms its language-unaware counterpart, yielding a 27% absolute improvement and winning over 78% of games when playing against the built-in AI of Civilization II.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Semantic Parsing on Freebase from Question-Answer Pairs (2013, background): “, 2013), playing computer games (Branavan et al., 2010; Branavan et al., 2011), following navigational instructions (Chen, 2012; Artzi and Zettlemoyer, 2013), and interacting in the real world via perception (Matuszek et al.”
Language Models as Knowledge Bases? (2019, neutral): “, 2019) or reinforcement learning (Branavan et al., 2011; Chevalier-Boisvert et al., 2018; Bahdanau et al., 2019; Luketina et al., 2019).”
Learning with Latent Language (2017, neutral): “Finally, natural language instructions available at concept-learning time (rather than language-learning time) have been used to provide side information to reinforcement learners about high-level strategy (Branavan et al., 2011), environments (Narasimhan et al.”
Joint Concept Learning and Semantic Parsing from Natural Language Explanations (2017, background): “Natural instructions have been studied in game playing frameworks (Branavan et al., 2012; Eisenstein et al., 2009).”
Deep Reinforcement Learning for Dialogue Generation (2016, background): “, 2016), executing instructions for Windows help (Branavan et al., 2011), or understanding dialogues that give navigation directions (Vogel and Jurafsky, 2010).”
A Roadmap Towards Machine Intelligence (2015, background): “…as learning to execute instructions in natural environments by interacting with humans (Thomason et al., 2015), or improving performance on real-life video-games by consulting the instruction manual (Branavan et al., 2012), that we would want our intelligent machines to also be able to carry out.”
A Survey of Reinforcement Learning Informed by Natural Language (2019, methodology): “[Branavan et al. , 2012] learn a Q -function that improves Monte-Carlo tree search planning for the ﬁrst few moves in Civilization II, a turn-based strategy game, while accessing the game’s natural language manual.”
Deep Reinforcement Learning with a Natural Language Action Space (2015, background): “There has also been interest in extracting textual knowledge to improve game control performance (Branavan et al., 2011), and mapping text instructions to sequences of executable actions (Branavan et al.”
Alignment-Based Compositional Semantics for Instruction Following (2015, methodology): “Indeed, previous work (Branavan et al., 2011; Narasimhan et al., 2015) uses aspects of both to solve a different class of gameplay problems.”
Vision-Based Navigation With Language-Based Assistance via Imitation Learning With Indirect Intervention (2018, background): “Recently, growing interest in grounded visual navigation from natural language is facilitated by the development of more realistic and complex simulation environments [2, 35, 9, 46, 59, 58, 51] in place of simple toy environments [5, 12, 39, 8, 7].”
PixL2R: Guiding Reinforcement Learning Using Natural Language by Mapping Pixels to Rewards (2020, methodology): “Branavan et al. [13] use a game manual to speed up learning, wherein the most relevant sentence from the manual is found for the current state using a log-linear model, and features are extracted from the sentence to augment the state representation.”
Improving Intrinsic Exploration with Language Abstractions (2022, background): “, 2019; Ammanabrolu and Riedl, 2019), read external knowledge sources (Branavan et al., 2012; Zhong et al., 2020; Hanjie et al., 2021), or use language for hierarchical planning (Andreas et al.”
Deep Transfer in Reinforcement Learning by Language Grounding (2017, unknown): “…include interpreting help documentation (Branavan et al., 2010), instruction following (Vogel and Jurafsky, 2010; Kol-lar et al., 2010; Artzi and Zettlemoyer, 2013; Ma-tuszek et al., 2013; Andreas and Klein, 2015) and learning to play computer games (Branavan et al., 2011; Narasimhan et al., 2015).”
Natural Language Communication with Robots (2016, background): “Recently the connection between less formulaic language and simple actions has been explored successfully in the context of simulated worlds (Branavan et al., 2009; Goldwasser and Roth, 2011; Branavan et al., 2011; Artzi and Zettlemoyer, 2013; Andreas and Klein, 2015) and videos (Malmaud et al.”
Towards a Dataset for Human Computer Communication via Grounded Language Acquisition (2016, background): “…between less formulaic language and simple actions has been explored successfully in the context of simulated worlds (Branavan et al. 2009; Goldwasser and Roth 2011; Branavan, Silver, and Barzilay 2011; Artzi and Zettlemoyer 2013) and videos (Malmaud et al. 2015; Venugopalan et al. 2015).”
Bringing Machine Learning and Compositional Semantics Together (2015, background): “…et al. 2013), interpreting natural language for performing programming tasks (Kushman & Barzilay 2013; Lei et al. 2013), playing computer games (Branavan et al. 2010, 2011) and interacting in the real world via perception (Matuszek et al. 2012a; Tellex et al. 2011; Krishnamurthy & Kollar 2013).”
RTFM: Generalising to Novel Environment Dynamics via Reading (2019, background): “…RL (see Luketina et al. (2019) for a recent survey) are limited to scenarios in which language speciﬁes the goal for some ﬁxed environment dynamics (Branavan et al., 2011; Hermann et al., 2017; Bahdanau et al., 2019; Fried et al., 2018; Co-Reyes et al., 2019), or the dynamics of the environment…”
Cognitive Architectures for Language Agents (2023, background): “In RL, “reading to learn” approaches (Branavan et al., 2012; Narasimhan et al., 2018; Hanjie et al., 2021; Zhong et al., 2021) leverage game manuals and facts as a semantic memory to affect the policy.”