BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments

By Siddharth Srivastava, Chengshu Li, Michael Lingelbach, Roberto Mart'in-Mart'in, Fei Xia, Kent Vainio, Zheng Lian, Cem Gokmen, Shyamal Buch, C. Karen Liu and 4 more. 2021

255 citations recorded by the source corpus.

We introduce BEHAVIOR, a benchmark for embodied AI with 100 activities in simulation, spanning a range of everyday household chores such as cleaning, maintenance, and food preparation. These activities are designed to be realistic, diverse, and complex, aiming to reproduce the challenges that agents must face in the real world. Building such a benchmark poses three fundamental difficulties for each activity: definition (it can differ by time, place, or person), instantiation in a simulator, and evaluation. BEHAVIOR addresses these with three innovations. First, we propose an object-centric, predicate logic-based description language for expressing an activity's initial and goal conditions, enabling generation of diverse instances for any activity. Second, we identify the simulator-agnostic features required by an underlying environment to support BEHAVIOR, and demonstrate its realization in one such simulator. Third, we introduce a set of metrics to measure task progress and efficiency, absolute and relative to human demonstrators. We include 500 human demonstrations in virtual reality (VR) to serve as the human ground truth. Our experiments demonstrate that even state of the art embodied AI solutions struggle with the level of realism, diversity, and complexity imposed by the activities in our benchmark. We make BEHAVIOR publicly available at behavior.stanford.edu to facilitate and calibrate the development of new embodied AI solutions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering, Evaluation & Benchmarks, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Evaluation & Benchmarks

In Robotics & Control

Citations in the corpus

Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, methodology): “These were developed to test various aspects of SayCan and were inspired by crowd sourcing via Amazon Mechanical Turk and in-person kitchen users, as well as benchmarks such as ALFRED [17] and BEHAVIOR [18].”
BEHAVIOR-1K: A Benchmark for Embodied AI with 1, 000 Everyday Activities and Realistic Simulation (2022, methodology): “The most similar benchmark to us is the previous generation BEHAVIOR-100 [27].”
ProcTHOR: Large-Scale Embodied AI Using Procedural Generation (2022, background): “For instance, in embodied AI, the houses may be used for navigation [58, 89, 118, 132, 117, 128, 74, 130], multi-agent interaction [51, 52, 2], rearrangement and interaction [115, 36, 39, 23, 106], manipulation [33, 86, 32, 122], Sim2Real transfer [27, 54, 66], embodied vision-andlanguage [105, 87, ”
ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills (2023, background): “Many benchmarks bypass critical challenges as a trade-off, by either adopting abstract grasp (Ehsani et al., 2021; Szot et al., 2021; Srivastava et al., 2022) or including few object-level variations (Zhu et al.”
A Survey of Large Language Models (2023, unknown): “BEHAVIOR [384] focuses on rearrangement tasks in simulation environments and requires LLMs to generate complex solutions, e.”
A Survey on Vision--Language--Action Models for Embodied AI (2024, unknown): “Table adapted from [188], [189].”
TidyBot: Personalized Robot Assistance with Large Language Models (2023, background): “For household cleanup in particular, the object rearrangement task [10], [15], [12], [16], [17] requires an embodied agent to pick up and move objects so as to bring the environment into a specified state.”
Advances, challenges and opportunities in creating data for trustworthy AI (2022, background): “In robotics, although real-world challenges provide the ultimate testbed, high-fidelity simulated environments are widely used to enable faster, safer learning on complex and long-horizon tasks 36,37 .”
Broadly-Exploring, Local-Policy Trees for Long-Horizon Task Planning (2020, methodology): “These methods then plan over them, for example, via trees or optimization-based methods [11, 12, 13, 14, 15, 16, 17].”
Evaluating Real-World Robot Manipulation Policies in Simulation (2024, unknown): “That is, if one policy performs better than another in real-world 1] , higher is better) for assessing the correlation between policy performances in real-world and simulation, as well as the overall quality of simulated evaluation pipelines.”
LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning (2023, unknown): “Then, the details about µ 0 are generated in the PDDL language [43, 63].”
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge (2022, methodology): “Prior works include maze-style worlds [121, 129, 61], purely text-based game [69], grid worlds [21, 16], browser/GUI-based environments [108, 124], and indoor simulators for robotics [1, 107, 114, 34, 110, 99, 89].”
VIMA: General Robot Manipulation with Multimodal Prompts (2022, methodology): “However, we envision future works can combine this formulation with more physically realistic simulators such as Zhu et al. (2020), Srivastava et al. (2021), and Mittal et al. (2023).”