Language Understanding for Text-based Games using Deep Reinforcement Learning

By Karthik Narasimhan, Tejas D. Kulkarni, Regina Barzilay. 2015

386 citations recorded by the source corpus.

In this paper, we consider the task of learning control policies for text-based games. In these games, all interactions in the virtual world are through text and the underlying state is not observed. The resulting language barrier makes such environments challenging for automatic game players. We employ a deep reinforcement learning framework to jointly learn state representations and action policies using game rewards as feedback. This framework enables us to map text descriptions into vector representations that capture the semantics of the game states. We evaluate our approach on two game worlds, comparing against baselines using bag-ofwords and bag-of-bigrams for state representations. Our algorithm outperforms the baselines on both worlds demonstrating the importance of learning expressive representations. 1

Subfields: Artificial Intelligence, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, background): “Finally, techniques which combine natural language with reinforcement learning environments (Narasimhan et al., 2015) have demonstrated exciting emergent behaviors such as systematically accomplishing zero-shot tasks (Hill et al., 2019).”
Coarse-to-Fine Question Answering for Long Documents (2016, background): “Reinforcement learning recently gained popularity in tasks such as co-reference resolution (Clark and Manning, 2016), information extraction (Narasimhan et al., 2016), semantic parsing (Andreas et al., 2016) and textual games (Narasimhan et al., 2015; He et al., 2016).”
Grounded Language Learning in a Simulated 3D World (2017, background): “Other recently-proposed frameworks for interactive language learning involve unimodal (text-only) settings (Narasimhan et al., 2015; Mikolov et al., 2015).”
ShapeWorld - A new test methodology for multimodal language understanding (2017, methodology): “A similar approach is taken by Narasimhan et al. (2015), but here the simulation is more complex, comprising a text-based role-playing game.”
Dialog-based Language Learning (2016, methodology): “Nevertheless, reinforcement learning ideas have been used to good effect for other tasks as well, such as understanding text adventure games [15], image captioning [31], machine translation and summarization [18].”
Sample Efficient Actor-Critic with Experience Replay (2016, methodology): “Replay is actually a valuable tool for improving sample efficiency and, as we will see in our experiments, state-of-the-art deep Q-learning methods (Schaul et al., 2016; Wang et al., 2016) have been up to this point the most sample efficient techniques on Atari by a significant margin.”
Alignment of Language Agents (2021, background): “Some examples of language agents we consider are generatively trained LLMs, such as Brown et al. (2020) and Radford et al. (2018, 2019), and RL agents in textbased games, such as Narasimhan et al. (2015).”
Pre-Trained Language Models for Interactive Decision-Making (2022, neutral): “A large body of recent work has used LM pre-training to scaffold interactive decision-making in text-based games (Yuan et al., 2018; Narasimhan et al., 2015; Ammanabrolu & Riedl, 2018; Côté et al., 2018; Yao et al., 2020; 2021) in which both observations and actions are encoded as text.”
Keep CALM and Explore: Language Models for Action Generation in Text-based Games (2020, background): “While several reinforcement learning (RL) models have been proposed recently (Narasimhan et al., 2015; He et al., 2015; Hausknecht et al., 2019a; Ammanabrolu and Riedl, 2019), combinatorially large action spaces continue to make these games challenging for these approaches.”
Prioritized Experience Replay (2015, background): “A recent paper introduced a form of re-sampling in the context of deep RL with experience replay (Narasimhan et al., 2015); the method separates experience into two buckets, one for positive and one for negative rewards, and then picks a ﬁxed fraction from each to replay.”
Deep Reinforcement Learning for Dialogue Generation (2016, background): “Also relevant is prior work on reinforcement learning for language understanding - including learning from delayed reward signals by playing text-based games (Narasimhan et al., 2015; He et al., 2016), executing instructions for Windows help (Branavan et al.”
Neural Approaches to Conversational AI (2018, positive): “Recent advances for solving text games, such as handling natural-language actions (Narasimhan et al., 2015; He et al., 2016; Côté et al., 2018) and interpretable policies (Chen et al.”
Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, methodology): “An alternative method is to use two separate experience buckets, one for positive and one for negative rewards [40].”
Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning (2017, background): “, 2017) or in text-based games (Narasimhan et al., 2015) via reinforcement learning.”
Online Contrastive Divergence with Generative Replay: Experience Replay without Storing Data (2016, background): “Narasimhan et al (2015) proposed a form of re-sampling in the context of DRL, separating the learner experience into two parts: one for the positive rewards and one for the negative.”
Batch Policy Gradient Methods for Improving Neural Conversation Models (2017, background): “Recently there has been a surge of interest in deep learning approaches to reinforcement learning, many of them adopting Q-learning, e.g. (He et al., 2015; Mnih et al., 2013; Narasimhan et al., 2015).”