ManipulaTHOR: A Framework for Visual Object Manipulation

By Kiana Ehsani, Winson Han, Alvaro Herrasti, Eli VanderBilt, Luca Weihs, Eric Kolve, Aniruddha Kembhavi, Roozbeh Mottaghi. 2021

164 citations recorded by the source corpus.

The domain of Embodied AI has recently witnessed substantial progress, particularly in navigating agents within their environments. These early successes have laid the building blocks for the community to tackle tasks that require agents to actively interact with objects in their environment. Object manipulation is an established research domain within the robotics community and poses several challenges including manipulator motion, grasping and long-horizon planning, particularly when dealing with oft-overlooked practical setups involving visually rich and complex scenes, manipulation using mobile agents (as opposed to tabletop manipulation), and generalization to unseen environments and objects. We propose a framework for object manipulation built upon the physics-enabled, visually rich AI2-THOR framework and present a new challenge to the Embodied AI community known as ArmPointNav. This task extends the popular point navigation task [2] to object manipulation and offers new challenges including 3D obstacle avoidance, manipulating objects in the presence of occlusion, and multi-object manipulation that necessitates long term planning. Popular learning paradigms that are successful on PointNav challenges show promise, but leave a large room for improvement.

Subfields: Computer Vision and Pattern Recognition, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Robotics & Control

Citations in the corpus

Habitat 2.0: Training Home Assistants to Rearrange their Habitat (2021, background): “ManipulaTHOR [34] is perhaps the most similar to our work.”
Continuous Scene Representations for Embodied AI (2022, background): “Recently, various embodied tasks such as navigation [5,8,37,54,56,63], instruction following [1,47], embodied question answering [15, 23], object manipulation [18,21,22,46], and room rearrangement [4,52] have been of interest.”
Evaluating Real-World Robot Manipulation Policies in Simulation (2024, unknown): “That is, if one policy performs better than another in real-world 1] , higher is better) for assessing the correlation between policy performances in real-world and simulation, as well as the overall quality of simulated evaluation pipelines.”
A Survey on Vision--Language--Action Models for Embodied AI (2024, unknown): “Versions: [197], [198] Benchmarks: ALFRED, RoPOR [199] VirtualHome [200] 7”
Instruction-driven history-aware policies for robotic manipulations (2022, background): “The emergence of robotic simulators, such as Gym [52], manipulaTHOR [53], dm control [54], Sapien [55], CausalWorld [56], and RL-Bench [19], also greatly accelerated the development of manipulation methods.”
VIMA: General Robot Manipulation with Multimodal Prompts (2022, background): “AI2-THOR (Ehsani et al., 2021; Deitke et al., 2022) is a framework that supports visual object manipulation and procedural generation of environments.”
iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks (2021, unknown): “Recent years, we have seen the emergence of many simulation environments for robotics and 22 embodied AI research [1, 2, 3, 4, 5, 6, 7, 8, 9, 10].”
CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents (2021, methodology): “We also note that recent work using AI2-THOR has done evaluations with object manipulation tasks (Shridhar et al., 2020; Batra et al., 2020; Ehsani et al., 2021), paving the way for more complex action spaces.”
Simple but Effective: CLIP Embeddings for Embodied AI (2021, background): “Simulators are photo-realistic, simulate physics, and enable interaction between agents and their environments [7, 8, 15, 16, 25, 26, 30, 36].”
ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills (2023, methodology): “Many benchmarks bypass critical challenges as a trade-off, by either adopting abstract grasp (Ehsani et al., 2021; Szot et al., 2021; Srivastava et al., 2022) or including few object-level variations (Zhu et al., 2020; Yu et al., 2020; James et al., 2020).”
Online Continual Learning For Interactive Instruction Following Agents (2024, background): “…substantial attention, and notable advancements have been made in various tasks (Anderson et al., 2018; Krantz et al., 2020; Jain et al., 2019; Savva et al., 2019; Deitke et al., 2020; Weihs et al., 2021; Ehsani et al., 2021; Shridhar et al., 2020; Pad-makumar et al., 2022; Gao et al., 2022).”
BEHAVIOR-1K: A Benchmark for Embodied AI with 1, 000 Everyday Activities and Realistic Simulation (2022, methodology): “To facilitate our experiments, we adopt an assistive pick primitive that creates a rigid connection between the object and the gripper if grasping is requested when all ﬁngers are in contact with the object, a stricter form of StickyMitten used in prior works [26, 63, 64].”
RT-1: Robotics Transformer for Real-World Control at Scale (2022, methodology): “via reinforcement-learning [32, 26, 67, 37, 65, 20, 18, 11, 69], and recently through foundation models [3, 38, 66].”
Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments (2023, background): “While recent frameworks [7], [11], [12] include physically-based renderers, they mainly support camera-based sensors (RGB, depth).”