MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge

By Linxi (Jim) Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andy Tang, De-An Huang, Yuke Zhu, Anima Anandkumar. Neural Information Processing Systems, 2022

597 citations recorded by the source corpus.

Autonomous agents have made great strides in specialist domains like Atari games and Go. However, they typically learn tabula rasa in isolated environments with limited and manually conceived objectives, thus failing to generalize across a wide spectrum of tasks and capabilities. Inspired by how humans continually learn and adapt in the open world, we advocate a trinity of ingredients for building generalist agents: 1) an environment that supports a multitude of tasks and goals, 2) a large-scale database of multimodal knowledge, and 3) a flexible and scalable agent architecture. We introduce MineDojo, a new framework built on the popular Minecraft game that features a simulation suite with thousands of diverse open-ended tasks and an internet-scale knowledge base with Minecraft videos, tutorials, wiki pages, and forum discussions. Using MineDojo's data, we propose a novel agent learning algorithm that leverages large pre-trained video-language models as a learned reward function. Our agent is able to solve a variety of open-ended tasks specified in free-form language without any manually designed dense shaping reward. We open-source the simulation suite, knowledge bases, algorithm implementation, and pretrained models (https://minedojo.org) to promote research towards the goal of generally capable embodied agents.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/32c9b3859086d15184989454eb878638659e64c6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

An Embodied Generalist Agent in 3D World (2023, unknown): “…grounded reasoning and task planning in the real world (Ahn et al., 2022; Huang et al., 2022b), skill generalization in open-world environment (Fan et al., 2022; Cai et al., 2023a; Wang et al., 2023d;a; Cai et al., 2023b; Gong et al., 2023b), general robotic manipulation (Brohan et al., 2022;…”
Human-Timescale Adaptation in an Open-Ended Task Space (2023, background): “…complex tasks, while Minecraft (Johnson et al., 2016) has been proposed as a challenge for Open-Endedness and RL (Kanervisto et al., 2022; Fan et al., 2022; Grbic et al., 2021), but unlike XLand it does not admit control of the full simulation stack, thereby limiting the smoothness of…”
Foundation models in robotics: Applications, challenges, and the future (2023, unknown): “MineDojo [71] is a framework for developing generalist agents in the game of Minecraft.”
Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks (2023, methodology): “Another 3D simulation library with a similar purpose is DeepMind Lab [2], but given that the game engine is written in C and the levels are written using Lua, there is a steep learning curve for customizing it.”
Voyager: An Open-Ended Embodied Agent with Large Language Models (2023, methodology): “MineDojo [23] and VPT [8] utilize YouTube videos for large-scale pre-training.”
Eureka: Human-Level Reward Design via Coding Large Language Models (2023, unknown): “Recent works have also proposed using pretrained foundation models that can produce reward functions for new tasks (Ma et al., 2022; 2023; Fan et al., 2022; Du et al., 2023a; Karamcheti et al., 2023; Du et al., 2023b; Kwon et al., 2023).”
Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models (2023, methodology): “We evaluate internal reasoning strategies by removing actions and observations from the context, corresponding to CoT (Wei et al., 2022) and its variants, CoT-SC (Wang et al., 2022), ToT (Yao et al., 2023a), and RAP (Hao et al., 2023).”
Foundation Models Defining a New Era in Vision: A Survey and Outlook (2025, unknown): “Furthermore, Mine-Dojo [158] gathers and trains agents on Minecraft data, while VOYAGER [159] is developed for lifelong learning based on LLM.”
MLVU: A Comprehensive Benchmark for Multi-Task Long Video Understanding (2024, unknown): “Additionally, it comprises 65 game videos from the MineDojo dataset [8], 200 surveillance videos from the UCF-Crime dataset [37], and 100 ego-centric videos from the Ego4D dataset [11].”
Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos (2022, methodology): “We hope future work will combine VPT with methods that can generate more generic reward functions, e.g. natural language based reward functions as proposed in MineDojo 70 (released after this paper).”
Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory (2023, methodology): “Some previous works have adopted knowledge distillation [24] and curriculum learning [11], while recent works [3, 5] tried to construct a language-conditioned multi-task agent via feeding the goal description embedding into the model.”
WebArena: A Realistic Web Environment for Building Autonomous Agents (2023, methodology): “(Kolve et al., 2017; Shridhar et al., 2020; Puig et al., 2018) and extends to gaming environments (Fan et al., 2022; Küttler et al., 2020), where the environment mechanisms often diverge from human objectives.”
Language Models can Solve Computer Tasks (2023, background): “Our agent’s action space, mainly restricted to clicks and typing, limits its web navigation capabilities.”
Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents (2023, methodology): “We report the final success rate of our method (DEP) with different selector implementations, including using a fixed sequence of goals, a random sequence of goals, and selecting a goal based on MineCLIP [10], CLIP [33], and our horizon-predictive Selector (HPS).”
LVLM-EHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models (2023, unknown): “First, LLaVA exhibits comparable performance to other LVLMs, despite being trained on the smallest dataset for both pre-training and instruction-tuning.”
Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models (2022, methodology): “MineCLIP [18] ﬁne-tunes CLIP [39] encoders using a contrastive loss on a large ofﬂine dataset of Minecraft videos and optimizes a language-conditioned control policy on top of the ﬁnetuned CLIP representations through online RL. LOReL [36] learns a reward function from ofﬂine robot datasets with…”
VIMA: General Robot Manipulation with Multimodal Prompts (2022, unknown): “MineDojo (Fan et al., 2022) and Ego4D (Grauman et al.”
AgentBench: Evaluating LLMs as Agents (2023, methodology): “, MineDojo [22]) as environments but mostly require multi-modal capabilities beyond existing LLMs.”