Keep CALM and Explore: Language Models for Action Generation in Text-based Games

By Shunyu Yao, Rohan Rao, Matthew J. Hausknecht, Karthik Narasimhan. 2020

156 citations recorded by the source corpus.

Text-based games present a unique challenge for autonomous agents to operate in natural language and handle enormous action spaces. In this paper, we propose the Contextual Action Language Model (CALM) to generate a compact set of action candidates at each game state. Our key insight is to train language models on human gameplay, where people demonstrate linguistic priors and a general game sense for promising actions conditioned on game history. We combine CALM with a reinforcement learning agent which re-ranks the generated action candidates to maximize in-game rewards. We evaluate our approach using the Jericho benchmark, on games unseen by CALM during training. Our method obtains a 69% relative improvement in average game score over the previous state-of-the-art model. Surprisingly, on half of these games, CALM is competitive with or better than other models that have access to ground truth admissible actions. Code and data are available at this https URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

ReAct: Synergizing Reasoning and Acting in Language Models (2022, background): “On the other hand, recent work has explored the use of pre-trained language models for planning and acting in interactive environments (Ahn et al., 2022; Nakano et al., 2021; Yao et al., 2020; Huang et al., 2022a), with a focus on predicting actions via language priors.”
WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents (2022, methodology): “These include: 1) generation of good search queries [22, 59] and reformulation [37, 51], 2) strategic exploration for navigating through the website [55, 56, 29], 3) robust language understanding for textual state and action spaces [3, 7, 17, 44], and 4) long-term”
Pre-Trained Language Models for Interactive Decision-Making (2022, neutral): “A large body of recent work has used LM pre-training to scaffold interactive decision-making in text-based games (Yuan et al., 2018; Narasimhan et al., 2015; Ammanabrolu & Riedl, 2018; Côté et al., 2018; Yao et al., 2020; 2021) in which both observations and actions are encoded as text.”
Language Models are Few-Shot Butlers (2021, methodology): “(1) As noted by Yao et al. (2020), where a j is the j-th token generated in action a of length m .”
Reading and Acting while Blindfolded: The Need for Semantics in Text Game Agents (2021, background): “A valid action handicap (Hausknecht et al., 2020) or a language model (Yao et al., 2020) can be used to generate a reduced action space for efficient exploration.”
Understanding the planning of LLM agents: A survey (2024, methodology): “CALM [ Yao et al. , 2020a ] proposed an early approach that combines a language model with an RL-based neural planner.”
Aligning to Social Norms and Values in Interactive Narratives (2022, methodology): “CALM is developed by Yao et al. (2020), this agent only uses the ClubFloyd dataset to train its action candidate generator without any value prior and uses the DRRN (He et al., 2016) architecture otherwise.”
Multi-Stage Episodic Control for Strategic Exploration in Text Games (2022, unknown): “Recently, Yao et al. (2021) used inverse dynamics to improve exploration and Yao et al. (2020) used a language model to generate action candidates that guide exploration.”
History Compression via Language Models in Reinforcement Learning (2022, methodology): “In text-based games, PLTs have been used for enhancing exploration (Yao et al., 2020).”
Learning Natural Language Generation with Truncated Reinforcement Learning (2022, methodology): “Besides, CALM is only evaluated on a vocabulary of 697 tokens, and on 4-words action sequences.”
Cognitive Architectures for Language Agents (2023, background): “…et al., 2019) would not support LLM agents — indeed, work at that time needed to combine GPT-2 with reinforcement learning for action generation (Yao et al., 2020); GPT-3 (Brown et al., 2020) unlocked flexible few-shot and zero-shot reasoning for NLP tasks; while only GPT-4 (OpenAI, 2023a)…”
AgentBench: Evaluating LLMs as Agents (2023, background): “However, it also leads to a common problem for LLM agents to generate invalid actions [97].”
Behavior Cloned Transformers are Neurosymbolic Reasoners (2022, background): “Because text games require interactive multistep reasoning, agents have most commonly been modelled using reinforcement learning (e.g. He et al., 2016; Zahavy et al., 2018; Yao et al., 2020), though overall performance on most environments remains low (see Jansen, 2022; Osborne et al.”