Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory

By Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, Yu Qiao and 2 more. arXiv.org, 2023

345 citations recorded by the source corpus.

The captivating realm of Minecraft has attracted substantial research interest in recent years, serving as a rich platform for developing intelligent agents capable of functioning in open-world environments. However, the current research landscape predominantly focuses on specific objectives, such as the popular"ObtainDiamond"task, and has not yet shown effective generalization to a broader spectrum of tasks. Furthermore, the current leading success rate for the"ObtainDiamond"task stands at around 20%, highlighting the limitations of Reinforcement Learning (RL) based controllers used in existing methods. To tackle these challenges, we introduce Ghost in the Minecraft (GITM), a novel framework integrates Large Language Models (LLMs) with text-based knowledge and memory, aiming to create Generally Capable Agents (GCAs) in Minecraft. These agents, equipped with the logic and common sense capabilities of LLMs, can skillfully navigate complex, sparse-reward environments with text-based interactions. We develop a set of structured actions and leverage LLMs to generate action plans for the agents to execute. The resulting LLM-based agent markedly surpasses previous methods, achieving a remarkable improvement of +47.5% in success rate on the"ObtainDiamond"task, demonstrating superior robustness compared to traditional RL-based controllers. Notably, our agent is the first to procure all items in the Minecraft Overworld technology tree, demonstrating its extensive capabilities. GITM does not need any GPU for training, but a single CPU node with 32 CPU cores is enough. This research shows the potential of LLMs in developing capable agents for handling long-horizon, complex tasks and adapting to uncertainties in open-world environments. See the project website at https://github.com/OpenGVLab/GITM.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/c695c4e68561347564ea0daa50dc339dff73d8c5

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Ping Luo
2. Yu Qiao
3. Wenhai Wang
4. Tong Lu
5. Jifeng Dai
6. Jianfeng Gao
7. Chunhua Shen
8. Luc Van Gool

Citations in the corpus

• Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models (2023, methodology): “Similar work (Baker et al., 2022; Wang et al., 2023; Zhu et al., 2023) has also adapted LLM agents to complex multimodal games such as Minecraft (Guss et al.”
• Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks (2023, background): “Recent advancements have seen the creation of vision large language models (VLLMs) [3, 21, 56, 59, 62, 66, 100, 121, 128, 130, 136, 138, 141, 142, 148], which aim to enhance language models with the capability to process and interpret visual information.”
• Executable Code Actions Elicit Better LLM Agents (2024, methodology): “…trajectories CodeActInstruct for instruction tuning, and train CodeActAgent that is specially designed for seamless integration with Python and can execute sophisticated tasks (e.g., model training) leveraging existing Python packages and autonomously rectifying errors through self-debugging.”
• Empowering Biomedical Discovery with AI Agents (2024, neutral): “Internal memory is directly used for accomplishing zero-shot tasks [6, 7] while accessing external memory requires actions by the agent to fetch and integrate data into short-term memory for immediate use [149, 150].”
• A survey on large language model based autonomous agents (2023, neutral): “Above, we mainly discuss the internal designs of the memory module.”
• The rise and potential of large language model based agents: a survey (2025, neutral): “…to control low-level policies [163] like a robotic transformer [101, 281, 282], which takes images and instructions as inputs and produces control commands for the end effector and robotic arms as well as virtual embodied controllers due to high costs of robotic operators [16,89,90,139,271,283,284].”
• Large language models empowered agent-based modeling and simulation: a survey and perspectives (2023, neutral): “This segmentation leverages the LLM ’ s training corpus to recognize patterns and apply relevant knowledge ef fi ciently (Park et al., 2023; Sun et al., 2024); Wang et al., 2024a; Zhu et al., 2023c).”
• Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate (2023, neutral): “We modulate the level of “tit for tat” state outlined in Section 2 through appending natural language instructions to the debaters’ meta prompt.”
• Foundation models in robotics: Applications, challenges, and the future (2023, neutral): “We also explore potential future avenues to address some of these challenges.”
• From entity-centric to goal-oriented graphs: Enhancing LLM knowledge retrieval in minecraft (2025, unknown): “Two methods that are more closely related to ours are GITM (Zhu et al., 2023) and Optimus-1 (Li et al., 2024).”