Gibson Env: Real-World Perception for Embodied Agents
By Fei Xia, Amir Zamir, Zhi-Yang He, Alexander Sax, Jitendra Malik, Silvio Savarese. 2018
1,020 citations recorded by the source corpus.
Developing visual perception models for active agents and sensorimotor control in the physical world are cumbersome as existing algorithms are too slow to efficiently learn in real-time and robots are fragile and costly. This has given rise to learning-in-simulation which consequently casts a question on whether the results transfer to real-world. In this paper, we investigate developing real-world perception for active agents, propose Gibson Environment for this purpose, and showcase a set of perceptual tasks learned therein. Gibson is based upon virtualizing real spaces, rather than artificially designed ones, and currently includes over 1400 floor spaces from 572 full buildings. The main characteristics of Gibson are: I. being from the real-world and reflecting its semantic complexity, II. having an internal synthesis mechanism "Goggles" enabling deploying the trained models in real-world without needing domain adaptation, III. embodiment of agents and making them subject to constraints of physics and space.
Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Control and Systems Engineering, Robotics & Control
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Aerospace Engineering
- Jitendra Malik
- Abhinav Kumar Gupta
- Silvio Savarese
- Kristen Grauman
- Vladlen Koltun
- Andreas Geiger
- Saurabh Gupta
- Sergey Levine
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Jitendra Malik
- Trevor Darrell
- Dacheng Tao
- Xiaogang Wang
- Kristen Grauman
- Ming-Hsuan Yang
- Abhinav Kumar Gupta
In Control and Systems Engineering
- Sergey Levine
- Chelsea Finn
- Pieter Abbeel
- Abhinav Kumar Gupta
- Kristen Grauman
- Dieter Fox
- Silvio Savarese
- Li Fei-Fei
In Robotics & Control
- Sergey Levine
- Dhruv Batra
- Abhinav Kumar Gupta
- Silvio Savarese
- Vladlen Koltun
- Kristen Grauman
- Chelsea Finn
- Jitendra Malik
Citations in the corpus
- Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding (2020, background): “sion community has developed several photorealistic synthetic datasets and interactive simulation environments that have spurred rapid progress towards the goal of holistic indoor scene understanding [5,6,8,9,13,14,17, 20,21,23,29,31,34,35,37,41,42,43,46,49,56,57,58, 60,65,67,70,71,74,78,79]. Figure”
- A Survey of Embodied AI: From Simulators to Research Tasks (2021, methodology): “Most of the larger-scale navigation-based simulators, such as AI2-THOR, iGibson and Habitat-Sim, tend to have navigation, atomic action and ROS [19, 34, 23] which enable them to provide better control and manipulation of objects in the environment while performing tasks such as Point Navigation or O”
- Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning (2019, background): “Similarly, there exists a number of robotics simulation benchmarks including visual navigation [39, 40, 41, 42, 43], autonomous driving [44, 45, 46], grasping [47, 48, 49], single-task manipulation [50], among others.”
- Taskonomy: Disentangling Task Transfer Learning (2018, background): “meshes similar to [3, 98, 12] enabling us to programmatically compute the ground truth for many tasks without human labeling.”
- Cognitive Mapping and Planning for Visual Navigation (2017, background): “Numerous works such as (Chang et al. 2017; Dai et al. 2017; Ammirato et al. 2017) have collected large-scale datasets consisting of scans of real world environments, while (Savva et al. 2017; Wu et al. 2018; Xia et al. 2018) have built more sophisticated simulation environments based on such scans.”
- Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation (2018, background): “In the past two years, a plethora of tasks and evaluation protocols [32, 22, 34, 45, 2] have been proposed as summarized in [1].”
- On Evaluation of Embodied Navigation Agents (2018, methodology): “A number of simulation platforms for indoor environments have recently been proposed [4, 12, 16, 28, 30, 31].”
- The Replica Dataset: A Digital Replica of Indoor Spaces (2019, methodology): “ns with a total of 270 rooms. It is annotated with 13 object classes and 11 scene categories. The exact method of semantic annotation is not described except that it is done in 3D. The Gibson dataset [28] contains 572 buildings and includes the two aforementioned datasets. Only the meshes from the M”
- SAPIEN: A SimulAted Part-Based Interactive ENvironment (2020, methodology): “have long used simulated environments for tasks such as navigation [43, 58, 44, 1, 3, 56, 14, 58] and control [24, 42, 50, 10].”
- Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration (2020, background): “, 2017) Low Ego Yes No No No Gibson (Xia et al., 2018) Low Ego Yes No No No AI Habitat (Savva et al.”
- iGibson 1.0: A Simulation Environment for Interactive Tasks in Large Realistic Scenes (2020, methodology): “Despite its high rendering speed, Habitat uses the noninteractive assets from Gibson v1 [25] and Matterport [34]”
- SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities (2024, background): “This is partially due to the difficulties of obtaining spatial-information-rich embodied data or high-quality human annotations for 3D-aware queries.”
- CLIP on Wheels: Zero-Shot Object Navigation as Object Localization and Exploration (2022, background): “…open-ended exploration, many navigation tasks are “goal-conditioned,” where the agent needs to navigate to a given position (i.e., point goal [58, 72, 74, 67, 11, 30]), a view of the environment (i.e., image goal [78, 43, 56]), or a specified object category (i.e., object goal [69, 10, 9, 75,…”
- iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks (2021, background): “0 [1] Gibson [2] Habitat [5] Sapien [4] AI2Thor [15] ManipulaTHOR [42] VirtualH [16] TDW [7, 13]”