The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark Towards Physically Realistic Embodied AI

By Chuang Gan, Siyuan Zhou, Jeremy Schwartz, Seth Alter, Abhishek Bhandwaldar, D. Gutfreund, Daniel L. K. Yamins, James J. DiCarlo, Josh H. McDermott, Antonio Torralba and 1 more. 2021

95 citations recorded by the source corpus.

We introduce a visually-guided task-and-motion planning benchmark, which we call the ThreeDWorld Trans-port Challenge. In this challenge, an embodied agent is spawned randomly in a simulated physical home environment and required to transport a small set of objects scattered around the house with containers. We build this benchmark challenge using the ThreeDWorld simulation: a virtual 3D environment where all objects respond to physics, and a robot agent can be controlled using a fully physics-driven navigation and interaction API. We evaluate several existing agents on this benchmark. Experimental results suggest that: 1) a pure RL model struggles on this challenge; 2) state-of-the-art hierarchical planning-based agents can transport some objects but are still far from solving this task. We anticipate that this benchmark will empower researchers to develop more intelligent physics-aware robot learning algorithms.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

In Robotics & Control

Citations in the corpus

BEHAVIOR-1K: A Benchmark for Embodied AI with 1, 000 Everyday Activities and Realistic Simulation (2022, background): “On the other hand, household rearrangement benchmarks such as Habitat 2.0 HAB [26], TDW Transport [19], and SAPIEN ManiSkill [54, 55] support realistic action execution and accurate physics simulation for rigid bodies, but only include a handful of tasks.”
Curious Representation Learning for Embodied Intelligence (2021, background): “We hope our proposed framework inspires future work towards learning both better task-agnostic representations and transferring to more complex embodied tasks [17].”
Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language (2021, background): “Physical Scene Understanding Our work is also relevant to studies on physical scene understanding [7, 71, 64, 62, 8, 23, 24, 48, 37], most of which propose pure neural-network solutions without explicitly incorporating physics models.”
BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments (2021, unknown): “Closer to BEHAVIOR, a recent group of benchmarks has focused on rearrangement tasks [23–25] in realistic simulation environments with diverse scenes.”
iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks (2021, unknown): “This kinodynamic 25 simulation is sufficient for pick-and-place and rearrangement tasks [11, 12, 13, 14]; however, as the 26 field advances, researchers are taking on more diverse and complex tasks that cannot be performed 27 in these simulators, e.”
ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills (2023, methodology): “, 2020), we refer to its Transport Challenge (Gan et al., 2021) (8)Assets like boards with slots for AssemblingKits, feasible layouts of obstacles for AvoidObstacles, and target characters for Write are not included.”
Building Cooperative Embodied Agents Modularly with Large Language Models (2023, background): “Rule-based Hierarchical Planner(RHP) is adopted from the strong performing baseline in the original ThreeDWorld Transport Challenge, which is a Hierarchical Planner with a high-level planner based on heuristics rules and a low-level A-start-based planner to navigate with semantic map, using Frontier”
ProcTHOR: Large-Scale Embodied AI Using Procedural Generation (2022, background): “For instance, in embodied AI, the houses may be used for navigation [58, 89, 118, 132, 117, 128, 74, 130], multi-agent interaction [51, 52, 2], rearrangement and interaction [115, 36, 39, 23, 106], manipulation [33, 86, 32, 122], Sim2Real transfer [27, 54, 66], embodied vision-andlanguage [105, 87, ”
VIMA: General Robot Manipulation with Multimodal Prompts (2022, background): “…Dasari & Gupta, 2020; Aceituno et al., 2021; Zhao et al., 2022), rearrangement (Weihs et al., 2021; Szot et al., 2021; Liu et al., 2021; Ehsani et al., 2021; Gan et al., 2021; Stengel-Eskin et al., 2022), and reasoning (Gupta et al., 2019; Ahmed et al., 2021; Toyer et al., 2020; Lim et al., 2021).”