Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding

By Mike Roberts, Nathan Paczan. 2020

719 citations recorded by the source corpus.

For many fundamental scene understanding tasks, it is difficult or impossible to obtain per-pixel ground truth labels from real images. We address this challenge by introducing Hypersim, a photorealistic synthetic dataset for holistic indoor scene understanding. To create our dataset, we leverage a large repository of synthetic scenes created by professional artists, and we generate 77,400 images of 461 indoor scenes with detailed per-pixel labels and corresponding ground truth geometry. Our dataset: (1) relies exclusively on publicly available 3D assets; (2) includes complete scene geometry, material information, and lighting information for every scene; (3) includes dense per-pixel semantic instance segmentations and complete camera information for every image; and (4) factors every image into diffuse reflectance, diffuse illumination, and a non-diffuse residual term that captures view-dependent lighting effects.We analyze our dataset at the level of scenes, objects, and pixels, and we analyze costs in terms of money, computation time, and annotation effort. Remarkably, we find that it is possible to generate our entire dataset from scratch, for roughly half the cost of training a popular open-source natural language processing model. We also evaluate sim-to-real transfer performance on two real-world scene understanding tasks – semantic segmentation and 3D shape prediction – where we find that pre-training on our dataset significantly improves performance on both tasks, and achieves state-of-the-art performance on the most challenging Pix3D test set. All of our rendered image data, as well as all the code we used to generate our dataset and perform our experiments, is available online.

Subfields: Computer Vision and Pattern Recognition, Geology, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Geology

In Evaluation & Benchmarks

Citations in the corpus

Segment Anything (2023, background): “ADE20K[117] BBBC038v1[12] Cityscapes[25] DOORS[80] DRAM[24] EgoHOS[113] GTEA[34,63] Hypersim[86] IBD[17] iShape[111] LVIS[44] NDD20[100] NDISPark[22,23] OVIS[81] PPDLS[74] Plittersdorf[46] STREETS[91] TimberSeg[38] TrashCan[52] VISOR[28,27] WoodScape[112] PIDRay[104] ZeroWaste-f[6] Figure 8: Samples”
Multiview Compressive Coding for 3D Reconstruction (2023, methodology): “9a shows qualitative results on Hypersim [52].”
GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image (2024, methodology): “We train our model on three categories: 1)Indoor: Hyper-sim [49] is a photorealistic synthetic dataset with 461 indoor scenes.”
ABO: Dataset and Benchmarks for Real-World 3D Object Understanding (2021, background): “There also exist many high quality datasets specifically made for 3D scene understanding [48,50,52,58] rather than individual objects.”
Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation (2023, positive): “Turning Stable Diffusion into Marigold requires only synthetic RGB-D data (in our case, the Hypersim [37] and Virtual KITTI [7] datasets) and a few GPU days on a single consumer graphics card.”
Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans (2021, methodology): “The starter data was created from 7 mesh-based datasets: Indoor scene datasets: Replica [56], HyperSim [49], Taskonomy [72], Habitat-Matterport (HM3D) Aerial/outdoor datasets: BlendedMVG [69] Diagnostic/Structured datasets: CLEVR [28] Object-centric datsets: To provide object-centric views in…”
Depth Anything 3: Recovering the Visual Space from Any Views (2025, neutral): “…[63] 99950 Synthetic ArkitScenes [5] 4388 LiDAR BlendedMVS [115] 503 3D Recon Co3dv2 [68] 30616 Colmap DL3DV [53] 6379 Colmap HyperSim [69] 344 Synthetic MapFree [3] 921 Colmap MegaDepth [51] 268 Colmap MegaSynth [40] 6049 Synthetic MvsSynth [38] 121 Synthetic Objaverse [17] 505557…”
3D-FRONT: 3D Furnished Rooms with layOuts and semaNTics (2020, background): “Also, Hypersim [30] presented a photorealistic synthetic dataset for holistic indoor scene understanding, focusing on providing per-pixel depth and disentangled illuminance and reflectance properties over scene images designed by professional artists.”
VGGT: Visual Geometry Grounded Transformer (2025, neutral): “Hence, while the feed-forward mode of VGGT outperforms all previous alternatives (whether they are feed-forward or not), there is still room for improvement since post-optimization still brings beneﬁts.”
Depth Anything V2 (2024, positive): “Therefore, to satisfy real-world applications such as multi-view synthesis, we fine-tune our powerful encoder on Hypersim [58] and Virtual KITTI [9] synthetic datasets, for indoor and outdoor metric depth estimation, respectively.”
MultiMAE: Multi-modal Multi-task Masked Autoencoders (2022, methodology): “We further evaluate our models on semantic segmentation tasks on the ADE20K [80] (20’210 training images and 150 classes), NYUv2 [57] (795 training images and 40 classes), and Hypersim [52] (51’674 training images and 40 classes) datasets.”
Qwen3-VL Technical Report (2025, positive): “…et al., 2014; Mao et al., 2016), ODinW-13 (Li et al., 2022), CountBench (Paiss et al., 2023), ARKitScenes (Baruch et al., 2021), Hypersim (Roberts et al., 2021), SUN RGB-D (Song et al., 2015), ERQA (Team et al., 2025), VSIBench (Yang et al., 2025b), EmbSpatial (Du et al.,…”
SAM 2: Segment Anything in Images and Videos (2024, neutral): “For completeness, we list the 23 datasets: LVIS (Gupta et al., 2019), ADE20K (Zhou et al., 2019), Hypersim (Roberts et al., 2021), Cityscapes (Cordts et al., 2016), BBBC038v1 (Caicedo et al., 2019), DOORS (Pugliatti & Topputo, 2022), DRAM (Cohen et al., 2022), EgoHOS (Zhang et al., 2022), GTEA…”
Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data (2024, background): “With these results, we hope to offer more insights for future works that similarly aim to build a general Method SUN RGB-D [56] iBims-1 [29] HyperSim [48] Virtual monocular depth estimation system.”
ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth (2023, methodology): “Zero-shot transfer to the HyperSim dataset [35].”
OpenRooms: An Open Framework for Photorealistic Indoor Scene Datasets (2020, background): “[61] diffuse shading Scenenet [39] CGIntrinsic [34] diffuse shading InteriorNet [32] diffuse shading CG-PBR [45] phong shading InvIndoor [33] microfacet envmap 3D-Future [18] AI2-THOR [31] Structure3D [64] shading Hypersim [43] diffuse highlight OpenRooms microfacet envmap al.”