Do Embodied Agents Dream of Pixelated Sheep?: Embodied Decision Making using Language Guided World Modelling

By Kolby Nottingham, Prithviraj Ammanabrolu, Alane Suhr, Yejin Choi, Hannaneh Hajishirzi, Sameer Singh, Roy Fox. 2023

115 citations recorded by the source corpus.

Reinforcement learning (RL) agents typically learn tabula rasa, without prior knowledge of the world. However, if initialized with knowledge of high-level subgoals and transitions between subgoals, RL agents could utilize this Abstract World Model (AWM) for planning and exploration. We propose using few-shot large language models (LLMs) to hypothesize an AWM, that will be verified through world experience, to improve sample efficiency of RL agents. Our DECKARD agent applies LLM-guided exploration to item crafting in Minecraft in two phases: (1) the Dream phase where the agent uses an LLM to decompose a task into a sequence of subgoals, the hypothesized AWM; and (2) the Wake phase where the agent learns a modular policy for each subgoal and verifies or corrects the hypothesized AWM. Our method of hypothesizing an AWM with LLMs and then verifying the AWM based on agent experience not only increases sample efficiency over contemporary methods by an order of magnitude but is also robust to and corrects errors in the LLM, successfully blending noisy internet-scale information from LLMs with knowledge grounded in environment dynamics.

Subfields: LLM Agents & Tool Use, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In LLM Agents & Tool Use

In Reinforcement Learning

Citations in the corpus

PaLM-E: An Embodied Multimodal Language Model (2023, neutral): “, 2022c), generating world models (Nottingham et al., 2023; Zellers et al., 2021a), planning over graphs and maps (Shah et al.”
A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis (2023, methodology): “Nottingham et al. (2023) and Wang et al. (2023b) perform sequential closed-loop planning on MineCraft.”
Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning (2023, background): “Other works have aimed to improve the planning performance of LLMs through prompt engineering [60] or collecting various forms of feedback such as sensory information [51, 20, 34], human corrections [60], self-corrections [46] or information of unmet preconditions [42, 56].”
The rise and potential of large language model based agents: a survey (2025, unknown): “Some researchers use LLM to decompose high-level task instructions into a series of sub-goals [315], basic skill sequences [314], or fundamental keyboard/mouse operations [315], gradually assisting agents in exploring the open world.”
A survey on large language model based autonomous agents (2023, unknown): “Robotics & Embodied Artificial Intelligence: Recent works have developed more efficient reinforcement learning agents for robotics and embodied artificial intelligence [25, 160, 106, 143, 133, 161, 60, 142, 154, 28, 2].”
A Survey on Vision--Language--Action Models for Embodied AI (2024, unknown): “DECKARD [76] prompts LLM to generate abstract world models (AWMs) represented as directed acyclic graphs, specifically tailored for the task of item crafting in Minecraft.”
Language Models can Solve Computer Tasks (2023, methodology): “First, our experiments use LLMs on HTML code, but ideally methods based on multimodal foundation models [13, 41, 2, 35] will be able to take actions based on text, images, audio, and video as input [3, 15, 33, 55].”
Voyager: An Open-Ended Embodied Agent with Large Language Models (2023, background): “[54] Kolby Nottingham, Prithviraj Ammanabrolu, Alane Suhr, Yejin Choi, Hanna Hajishirzi, Sameer Singh, and Roy Fox.”