Counting to Explore and Generalize in Text-based Games

By Xingdi Yuan, Marc-Alexandre Côté, Alessandro Sordoni, Romain Laroche, Rémi Tachet des Combes, Matthew J. Hausknecht, Adam Trischler. 2018

76 citations recorded by the source corpus.

We propose a recurrent RL agent with an episodic exploration mechanism that helps discovering good policies in text-based game environments. We show promising results on a set of generated text-based games of varying difficulty where the goal is to collect a coin located at the end of a chain of rooms. In contrast to previous text-based RL approaches, we observe that our agent learns policies that generalize to unseen games of greater difficulty.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Pre-Trained Language Models for Interactive Decision-Making (2022, neutral): “A large body of recent work has used LM pre-training to scaffold interactive decision-making in text-based games (Yuan et al., 2018; Narasimhan et al., 2015; Ammanabrolu & Riedl, 2018; Côté et al., 2018; Yao et al., 2020; 2021) in which both observations and actions are encoded as text.”
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning (2020, methodology): “Following the training strategy used in the recurrent DQN literature (Hausknecht and Stone, 2015; Yuan et al., 2018), we use the ﬁrst 2 transitions to estimate the recurrent states, and the last 2 transitions for updating the model parameters.”
A Survey of Reinforcement Learning Informed by Natural Language (2019, background): “[DePristo and Zubek, 2001; Narasimhan et al. , 2015; Yuan et al. , 2018] observe that when the action space of the text game is constrained to verb-object pairs, decomposing the Q -function into separate parts for verb and object provides enough structure to make learning more tractable.”
Interactive Fiction Games: A Colossal Adventure (2019, background): “TextWorld domains have already proven suitable for reinforcement learning agents (Yuan et al. 2018) which were shown to be capable of learning on a set of environments and then generalizing to unseen ones at test time.”
Towards Solving Text-based Games by Producing Adaptive Action Spaces (2018, methodology): “Another model is inspired by Yuan et al. (2018b) who generates concatenated target sentences with orthogonally regularized separators.”
How to Motivate Your Dragon: Teaching Goal-Driven Agents to Speak and Act in Fantasy Worlds (2020, background): “(Côté et al., 2018) introduce TextWorld, a framework for procedurally generating text-based games via grammars, and (Yuan et al., 2018; Yin and May, 2019; Adolphs and Hofmann, 2019; Adhikari et al., 2020) build agents that operate in this environment—focusing on aspects such as efficient…”
Algorithmic Improvements for Deep Reinforcement Learning applied to Interactive Fiction (2019, methodology): “Our baseline agent is the LSTM-DRQN agent (Yuan et al. 2018) but with a different action representation.”
An Overview of Natural Language State Representation for Reinforcement Learning (2020, methodology): “Subsequent works followed the same approach (Ansari et al., 2018; Yuan et al., 2018; Jain et al., 2020) or used variants of continuous vectors built by neural networks (He et al., 2016; Foerster et al., 2016).”
Learning Dynamic Belief Graphs to Generalize on Text-Based Games (2020, methodology): “As baselines, we use our implementation of LSTM-DQN [29] and LSTM-DRQN [50], both of which use onlyOt as input.”
NAIL: A General Interactive Fiction Agent (2019, methodology): “Building on this idea, LSTM-DRQN (Yuan et al., 2018) added a second level recurrence over states, which was demonstrated to give the agent an ability to reason over locations visited in the past.”
One-Shot Learning from a Demonstration with Hierarchical Latent Language (2022, unknown): “Speciﬁcally, following the training strategy used in the recurrent DQN literature (Hausknecht and Stone, 2015; Yuan et al., 2018), we sample batches of transition sequences (of length 8), we use the ﬁrst 4 transitions to estimate the recurrent states, and the last 4 transitions for updating the…”
Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning (2018, background): “In addition to language understanding, successful play generally requires long-term memory, planning, exploration (Yuan et al., 2018), affordance extraction (Fulda et al., 2017), and common sense.”