Learning Dynamic Belief Graphs to Generalize on Text-Based Games

By Ashutosh Adhikari, Xingdi Yuan, Marc-Alexandre Côté, M. Zelinka, Marc-Antoine Rondeau, Romain Laroche, Pascal Poupart, Jian Tang, Adam Trischler, William L. Hamilton. 2020

88 citations recorded by the source corpus.

Playing text-based games requires skills in processing natural language and sequential decision making. Achieving human-level performance on text-based games remains an open challenge, and prior research has largely relied on hand-crafted structured representations and heuristics. In this work, we investigate how an agent can plan and generalize in text-based games using graph-structured representations learned end-to-end from raw text. We propose a novel graph-aided transformer agent (GATA) that infers and updates latent belief graphs during planning to enable effective action selection by capturing the underlying game dynamics. GATA is trained using a combination of reinforcement and self-supervised learning. Our work demonstrates that the learned graph-based representations help agents converge to better policies than their text-only counterparts and facilitate effective generalization across game configurations. Experiments on 500+ unique games from the TextWorld suite show that our best agent outperforms text-based baselines by an average of 24.2%.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

ALFWorld: Aligning Text and Embodied Environments for Interactive Learning (2020, background): “Prior text-based agents typically use a list of candidate commands from the game engine (Adhikari et al., 2020) or populate a list of command templates (Ammanabrolu and Hausknecht, 2020).”
Behavior Cloned Transformers are Neurosymbolic Reasoners (2022, background): “As a result, nearly all contemporary models (e.g. Ammanabrolu and Hausknecht, 2020; Adhikari et al., 2020; Murugesan et al., 2021) make use of the valid action aid (Hausknecht et al.”
Aligning to Social Norms and Values in Interactive Narratives (2022, background): “Recent works in this area have focused on tackling three primary challenges: (1) how to represent agent knowledge to effectively operate in partially observable environments (Adhikari et al., 2020; Sautier et al., 2020); (2) scaling RL algorithms to handle combinatorial natural language state-action”
Interactive Fiction Game Playing as Multi-Paragraph Reading Comprehension with Reinforcement Learning (2020, unknown): “Previous work mainly studies the text understanding and generation in parserbased or rule-based text game tasks, such as TextWorld platform (Côté et al., 2018) or custom domains (Narasimhan et al., 2015; He et al., 2016; Adhikari et al., 2020).”
Can Language Models Solve Graph Problems in Natural Language? (2023, background): “In the belief state graph in theory-of-mind [Adhikari et al., 2020, Ammanabrolu and Riedl, 2021], the final task is to determine the connectivity of nodes, which corresponds to the connectivity task.”
Situated Dialogue Learning through Procedural Environment Generation (2021, background): “They aid in the challenges of partial observability/knowledge representation (Ammanabrolu and Riedl, 2019; Adhikari et al., 2020; Sautier et al., 2020), combinatorial state-action spaces (Ammanabrolu and Hausknecht, 2020; Ammanabrolu et al.”
How to Motivate Your Dragon: Teaching Goal-Driven Agents to Speak and Act in Fantasy Worlds (2020, background): “…a framework for procedurally generating text-based games via grammars, and (Yuan et al., 2018; Yin and May, 2019; Adolphs and Hofmann, 2019; Adhikari et al., 2020) build agents that operate in this environment—focusing on aspects such as efficient exploration and zeroshot generalization to…”