Playing Text-Adventure Games with Graph-Based Deep Reinforcement Learning

By Prithviraj Ammanabrolu, Mark O. Riedl. 2018

132 citations recorded by the source corpus.

Text-based adventure games provide a platform on which to explore reinforcement learning in the context of a combinatorial action space, such as natural language. We present a deep reinforcement learning architecture that represents the game state as a knowledge graph which is learned during exploration. This graph is used to prune the action space, enabling more efficient exploration. The question of which action to take can be reduced to a question-answering task, a form of transfer learning that pre-trains certain parts of our architecture. In experiments using the TextWorld framework, we show that our proposed technique can learn a control policy faster than baseline alternatives. We have also open-sourced our code at https://github.com/rajammanabrolu/KG-DQN.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Pre-Trained Language Models for Interactive Decision-Making (2022, background): “Is this simply because LMs are effective models of relations between natural language descriptions of states and actions [1], or because they provide a more general framework for combinatorial generalization in decision-making? We hypothesize and investigate three possible factors to understand the ”
Keep CALM and Explore: Language Models for Action Generation in Text-based Games (2020, background): “While several reinforcement learning (RL) models have been proposed recently (Narasimhan et al., 2015; He et al., 2015; Hausknecht et al., 2019a; Ammanabrolu and Riedl, 2019), combinatorially large action spaces continue to make these games challenging for these approaches.”
Interactive Fiction Games: A Colossal Adventure (2019, background): “Ammanabrolu and Riedl (2019b) further expand on this work, exploring methods of transferring control policies in text-games, using knowledge graphs to seed an agent with useful commonsense knowledge and transfer knowledge between different games within a domain.”
Graph Constrained Reinforcement Learning for Natural Language Action Spaces (2020, methodology): “From the perspective of text-game playing agent and most in line with the spirit of our work, Ammanabrolu & Riedl (2019a) present the Knowledge Graph DQN or KG-DQN, an approach where a knowledge graph built during exploration is used as a state representation for a deep reinforcement learning based…”
A Long Way to Go: Investigating Length Correlations in RLHF (2023, unknown): “We focus on length as it is the most prevalent, but our experimental paradigm is applicable to any analysis of over-optimization in RLHF. Length control and length biases Techniques outside of RLHF for controlling length of NLP models have been explored (Kikuchi et al., 2016; Ficler & Goldberg, 2017”
Situated Dialogue Learning through Procedural Environment Generation (2021, background): “They aid in the challenges of partial observability/knowledge representation (Ammanabrolu and Riedl, 2019; Adhikari et al., 2020; Sautier et al., 2020), combinatorial state-action spaces (Ammanabrolu and Hausknecht, 2020; Ammanabrolu et al.”
Interactive Fiction Game Playing as Multi-Paragraph Reading Comprehension with Reinforcement Learning (2020, methodology): “We parameterize the observation action value function Q ( o, a = (cid:104) verb , arg 0 , arg 1 ; θ ) by utilizing the decomposition of the template actions and context-query contextualized representation in RC.”
Improving Intrinsic Exploration with Language Abstractions (2022, background): “, 2020), play text-based games (Côté et al., 2018; Urbanek et al., 2019; Ammanabrolu and Riedl, 2019), read external knowledge sources (Branavan et al.”
Multi-Stage Episodic Control for Strategic Exploration in Text Games (2022, unknown): “Second, these games have very sparse rewards (see Appendix A.1) and several bottleneck states (Ammanabrolu et al., 2020), making learning difﬁcult for RL agents that use the same policy for exploration and exploitation (He et al., 2016a; Hausknecht et al., 2020; Ammanabrolu & Riedl, 2019).”
NAIL: A General Interactive Fiction Agent (2019, background): “Finally, Ammanabrolu and Riedl (2018) .0008 You try to push past, but vines block your way.”
Algorithmic Improvements for Deep Reinforcement Learning applied to Interactive Fiction (2019, background): “Finally, Ammanabrolu and Riedl (2019) deal with partial observability by representing state as a knowledge graph and continuously updating it after every game step.”
What Would Jiminy Cricket Do? Towards Agents That Behave Morally (2021, methodology): “Later algorithms such as KG-DQN, KG-A2C, and GATA incorporate knowledge graphs to improve inductive biases (Ammanabrolu and Riedl, 2019; Ammanabrolu and Hausknecht, 2020; Adhikari et al., 2020; Ammanabrolu et al., 2020).”
Generating Interactive Worlds with Text (2019, background): “Ammanabrolu and Riedl (2019) represent a textbased adventure game as a graph and learn how to adventure within this world.”
Learning Dynamic Belief Graphs to Generalize on Text-Based Games (2020, background): “More recent approaches take advantage of the graph-like structure of textbased games by building knowledge graph (KG) representations of the game state: Ammanabrolu and Riedl [4], Ammanabrolu and Hausknecht [3], for example, use hand-crafted heuristics to populate a KG that feeds into a deep neural ”
Learning Natural Language Generation with Truncated Reinforcement Learning (2022, methodology): “(Ammanabrolu and Riedl, 2018) use the structure of language to prune the action space of text-based games, but within value-based algorithms, which are less scalable to large vocabularies.”
An Overview of Natural Language State Representation for Reinforcement Learning (2020, background): “Ammanabrolu & Riedl (2019) combined the embedded text (by a sliding BiL-STM) with an embedded knowledge graph built by the agent throughout the game.”
Goal-Directed Story Generation: Augmenting Generative Language Models with Reinforcement Learning (2021, background): “For example, knowledge graphs were shown to improve agent decision-making in interactive narrative text game playing domains (Ammanabrolu and Riedl, 2018; Ammanabrolu and Hausknecht, 2020; Ammanabrolu et al., 2020b; Ammanabrolu and Riedl, 2021).”
Toward Automated Quest Generation in Text-Adventure Games (2019, background): “…2010; He et al., 2016; Narasimhan et al., 2015; Fulda et al., 2017; Yang et al., 2018; Haroush et al., 2018; Cˆot´e et al., 2018; Tao et al., 2018; Ammanabrolu and Riedl, 2019a; Hausknecht et al., 2019a; Am-manabrolu and Riedl, 2019b; Hausknecht et al., 2019b), these works focus on creating…”