Deep Reinforcement Learning with a Natural Language Action Space

By Ji He, Jianshu Chen, Xiaodong He, Jianfeng Gao, Lihong Li, Li Deng, Mari Ostendorf. 2015

268 citations recorded by the source corpus.

This paper introduces a novel architecture for reinforcement learning with deep neural networks designed to handle state and action spaces characterized by natural language, as found in text-based games. Termed a deep reinforcement relevance network (DRRN), the architecture represents action and state spaces with separate embedding vectors, which are combined with an interaction function to approximate the Q-function in reinforcement learning. We evaluate the DRRN on two popular text games, showing superior performance over other deep Q-learning architectures. Experiments with paraphrased action descriptions show that the model is extracting meaning rather than simply memorizing strings of text.

Subfields: Artificial Intelligence, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Language Models are Few-Shot Butlers (2021, methodology): “CALM was combined with game-speciﬁc models trained with re-inforcement learning (He et al., 2016) to pick the best action candidate among CALM’s generations.”
Keep CALM and Explore: Language Models for Action Generation in Text-based Games (2020, methodology): “Figure 2: CALM combined with an RL agent – DRRN (He et al., 2015) – for gameplay.”
Deep Reinforcement Learning for Dialogue Generation (2016, background): “Also relevant is prior work on reinforcement learning for language understanding - including learning from delayed reward signals by playing text-based games (Narasimhan et al., 2015; He et al., 2016), executing instructions for Windows help (Branavan et al.”
Neural Approaches to Conversational AI (2018, neutral): “ys similar to a conversation, except that the scenarios are predeﬁned by the game designer. Recent advances for solving text games, such as handling natural-language actions (Narasimhan et al., 2015; He et al., 2016; Cotˆ ´e et al., 2018) and interpretable policies (Chen et al., 2017c) may inspire s”
Batch Policy Gradient Methods for Improving Neural Conversation Models (2017, background): “Although there are ways to increase the diversity of responses (Li et al., 2015), our focus is on encouraging the bot to meaningfully advance the conversation.”
RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, background): “Action representations have been used to solve natural language processing problems (He et al., 2016), enable generalization to unseen (Ayush et al.”
TextWorld: A Learning Environment for Text-based Games (2018, neutral): “Image from [He et al., 2015].”
Mapping Instructions and Visual Observations to Actions with Reinforcement Learning (2017, methodology): “Reinforcement learning with neural networks has been used for various NLP tasks, including text-based games (Narasimhan et al., 2015; He et al., 2016), information extraction (Narasimhan et al., 2016), co-reference resolution (Clark and Manning, 2016), and dialog (Li et al., 2016).”
Interactive Fiction Games: A Colossal Adventure (2019, methodology): “The Deep Reinforcement Relevance Network (DRRN) (He et al. 2016) is an algorithm for choice-based games that present a set of valid actions Avalid(s) at every game state s.”
Improving Information Extraction by Acquiring External Evidence with Reinforcement Learning (2016, methodology): “The DQN, in which the Q-function is approximated using a deep neural network, has been shown to learn better value functions than linear approximators (Narasimhan et al., 2015; He et al., 2015) and can capture non-linear interactions between the different pieces of information in our state.”
MathDQN: Solving Arithmetic Word Problems via Deep Reinforcement Learning (2018, background): “In previous efforts of deep reinforcement learning, such as information extraction (Narasimhan et al. 2016) and text game (He et al. 2016), states were represented as real-valued vector with fixed number of dimensions.”
Language as an Abstraction for Hierarchical Deep Reinforcement Learning (2019, background): “Another line of work focuses on RL for textual adventure games where the state is represented as language descriptions and the actions are either textual actions available at each state [23] or all possible actions [39] (even though not every action is applicable to all states).”
Graph Constrained Reinforcement Learning for Natural Language Action Spaces (2020, methodology): “The DRRN algorithm for choice-based games (He et al., 2016; Zelinka, 2018) estimates Q-Values for a particular action from a particular state.”
Deep Reinforcement Learning: An Overview (2017, methodology): “The authors represented both πθ andDR as deep neural networks, and found an optimal solution by repeatedly performing gradient updates on each of them. DR can be trained with supervised learning with a data set formed from traces from a current πθ and expert traces. For a fixed DR, an optimalπθ is s”
Interactive Fiction Game Playing as Multi-Paragraph Reading Comprehension with Reinforcement Learning (2020, background): “Deep Reinforcement Relevance Network (DRRN) [9] projects each action into a hidden space that matches the current state representation vector for action selection.”
Deep Reinforcement Learning in Large Discrete Action Spaces (2015, background): “Recent work extends Deep Q-Networks to ‘unbounded’ action spaces (He et al., 2015), effectively generating action representations for any action the environment provides, and picking the action that provides the highest Q.”
Towards Solving Text-based Games by Producing Adaptive Action Spaces (2018, background): “…that are possible in a given context (e.g., open the door if the context contains an unlocked door), and the second task is the reinforcement learning task of learning to act optimally in order to solve the game (Narasimhan et al., 2015; Zelinka, 2018; He et al., 2015; Haroush et al., 2018).”
An empirical investigation of the challenges of real-world reinforcement learning (2020, background): “He et al. (2015) present the Deep Reinforcement Relevance Network (DRRN) for evaluating continuous action spaces in text-based games.”