Algorithmic Improvements for Deep Reinforcement Learning applied to Interactive Fiction

By Vishal Jain, William Fedus, Hugo Larochelle, Doina Precup, Marc G. Bellemare. 2019

32 citations recorded by the source corpus.

Text-based games are a natural challenge domain for deep reinforcement learning algorithms. Their state and action spaces are combinatorially large, their reward function is sparse, and they are partially observable: the agent is informed of the consequences of its actions through textual feedback. In this paper we emphasize this latter point and consider the design of a deep reinforcement learning agent that can play from feedback alone. Our design recognizes and takes advantage of the structural characteristics of text-based games. We first propose a contextualisation mechanism, based on accumulated reward, which simplifies the learning problem and mitigates partial observability. We then study different methods that rely on the notion that most actions are ineffectual in any given situation, following Zahavy et al.'s idea of an admissible action. We evaluate these techniques in a series of text-based games of increasing difficulty based on the TextWorld framework, as well as the iconic game Zork. Empirically, we find that these techniques improve the performance of a baseline deep reinforcement learning agent applied to text-based games.

Subfields: Artificial Intelligence, Economics and Econometrics, Sociology and Political Science, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Economics and Econometrics

In Sociology and Political Science

In Reinforcement Learning

Citations in the corpus

Keep CALM and Explore: Language Models for Action Generation in Text-based Games (2020, background): “Even with small actions spaces (e.g. < 200 actions), approaches to filter inadmissible actions (Zahavy et al., 2018; Jain et al., 2019) led to faster learning convergence.”
Graph Constrained Reinforcement Learning for Natural Language Action Spaces (2020, background): “Several other authors also report scores at this threshold (Jain et al., 2019; Zahavy et al., 2018).”
Learning Dynamic Belief Graphs to Generalize on Text-Based Games (2020, background): “[20] explicitly condition an agent’s decisions on game feedback.”
An Overview of Natural Language State Representation for Reinforcement Learning (2020, methodology): “Subsequent works followed the same approach (Ansari et al., 2018; Yuan et al., 2018; Jain et al., 2020) or used variants of continuous vectors built by neural networks (He et al., 2016; Foerster et al., 2016).”