Graph Constrained Reinforcement Learning for Natural Language Action Spaces

By Prithviraj Ammanabrolu, Matthew J. Hausknecht. 2020

139 citations recorded by the source corpus.

Interactive Fiction games are text-based simulations in which an agent interacts with the world purely through natural language. They are ideal environments for studying how to extend reinforcement learning agents to meet the challenges of natural language understanding, partial observability, and action generation in combinatorially-large text-based action spaces. We present KG-A2C, an agent that builds a dynamic knowledge graph while exploring and generates actions using a template-based action space. We contend that the dual uses of the knowledge graph to reason about game state and to constrain natural language generation are the keys to scalable exploration of combinatorially large natural language actions. Results across a wide variety of IF games show that KG-A2C outperforms current IF agents despite the exponential increase in action space size.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (2022, methodology): “To address this issue, we introduce NLPO (Natural Language Policy Optimization), which is inspired by work on action elimination/invalid-action masking (Zahavy et al., 2018; Huang & Ontañón, 2020; Ammanabrolu & Hausknecht, 2020).”
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning (2020, background): “Prior text-based agents typically use a list of candidate commands from the game engine (Adhikari et al., 2020) or populate a list of command templates (Ammanabrolu and Hausknecht, 2020).”
Keep CALM and Explore: Language Models for Action Generation in Text-based Games (2020, methodology): “, 2019a) or handicaps provided by the learning environment (Hausknecht et al., 2019a; Ammanabrolu and Hausknecht, 2020) to circumvent these issues.”
What Would Jiminy Cricket Do? Towards Agents That Behave Morally (2021, methodology): “Later algorithms such as KG-DQN, KG-A2C, and GATA incorporate knowledge graphs to improve inductive biases (Ammanabrolu and Riedl, 2019; Ammanabrolu and Hausknecht, 2020; Adhikari et al., 2020; Ammanabrolu et al., 2020).”
Behavior Cloned Transformers are Neurosymbolic Reasoners (2022, background): “Overall performance is generally low, with agents currently solving only 30% of classic interactive fiction games such as Zork (Ammanabrolu and Hausknecht, 2020; Yao et al., 2021; Atzeni et al., 2022).”
A Metaverse: Taxonomy, Components, Applications, and Open Challenges (2022, unknown): “Knowledge Graph A2C (KG-A2C) is a scalable exploratory method for inferring game states in a template-based workspace using linguistic behaviors and dynamic knowledge graphs [160].”
Multi-Stage Episodic Control for Strategic Exploration in Text Games (2022, unknown): “While these approaches focus mainly on the issues of partial observability and language semantics, they all suffer from challenges due to the large action space and sparse rewards found in games from benchmarks like Jericho (Hausknecht et al., 2020).”
Situated Dialogue Learning through Procedural Environment Generation (2021, background): “, 2020), combinatorial state-action spaces (Ammanabrolu and Hausknecht, 2020; Ammanabrolu et al., 2020c), and commonsense reasoning (Ammanabrolu and Riedl, 2019; Murugesan et al.”
Toward Automated Quest Generation in Text-Adventure Games (2019, background): “Although there has been much work recently on textadventure gameplay (Bordes et al. 2010; He et al. 2016; Narasimhan, Kulkarni, and Barzilay 2015; Fulda et al. 2017; Yang et al. 2018; Haroush et al. 2018; Côté et al. 2018; Tao et al. 2018; Ammanabrolu and Riedl 2019a; 2019b; Hausknecht et al. 2019a;”
Reading and Acting while Blindfolded: The Need for Semantics in Text Game Agents (2021, background): “, 2020), knowledge graphs (Ammanabrolu and Hausknecht, 2020; Ammanabrolu et al., 2020; Adhikari et al., 2020), and reading comprehension systems (Guo et al.”
Learning Dynamic Belief Graphs to Generalize on Text-Based Games (2020, background): “2 For instance, Hausknecht et al. [16] show that a state-of-the-art model achieves a mere 2 .”
Interactive Fiction Game Playing as Multi-Paragraph Reading Comprehension with Reinforcement Learning (2020, methodology): “Previous approaches address this problem by building a representation over past observations (e.g., building a graph of objects, positions, and spatial relations) [3, 2].”
Goal-Directed Story Generation: Augmenting Generative Language Models with Reinforcement Learning (2021, background): “For example, knowledge graphs were shown to improve agent decision-making in interactive narrative text game playing domains (Ammanabrolu and Riedl, 2018; Ammanabrolu and Hausknecht, 2020; Ammanabrolu et al., 2020b; Ammanabrolu and Riedl, 2021).”
How to Motivate Your Dragon: Teaching Goal-Driven Agents to Speak and Act in Fantasy Worlds (2020, background): “This resulted in agents developed by works such as (Zahavy et al., 2018; Ammanabrolu and Hausknecht, 2020), aiming to learn to execute contextually relevant actions.”
Aligning to Social Norms and Values in Interactive Narratives (2022, methodology): “The ClubFloyd dataset (Yao et al., 2020; Ammanabrolu and Hausknecht, 2020) used to train the action candidate generators consists of human transcripts of the form Tk = (o0, a0, o1, a1.”