Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans

By Ainaz Eftekhar, Alexander Sax, Roman Bachmann, Jitendra Malik, Amir Zamir. IEEE International Conference on Computer Vision, 2021

448 citations recorded by the source corpus.

This paper introduces a pipeline to parametrically sample and render static multi-task vision datasets from comprehensive 3D scans from the real-world. In addition to enabling interesting lines of research, we show the tooling and generated data suffice to train robust vision models. Familiar architectures trained on a generated starter dataset reached state-of-the-art performance on multiple common vision tasks and benchmarks, despite having seen no benchmark or non-pipeline data. The depth estimation network outperforms MiDaS and the surface normal estimation network is the first to achieve human-level performance for in-the-wild surface normal estimation—at least according to one metric on the OASIS benchmark. The Dockerized pipeline with CLI, the (mostly python) code, PyTorch dataloaders for the generated data, the generated starter dataset, download scripts and other utilities are all available ${\color{Magenta}through}\;{\color{Magenta}our}\;{\color{Magenta}project}\;{\color{Magenta}website}$.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/f1a66139fa051370018a3539f15e60f728e437ca

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddings (2022, methodology): “The observation is processed by a ResNet-50 [23] 92 encoder, which is pretrained on the Omnidata Starter Dataset (OSD) [24] using self-supervised 93 learning (DINO [25]) following the pretraining recipe presented in OVRL [16].”
GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image (2024, methodology): “Hereby, We choose both SoTA in-domain (EENSU [1]) and zero-shot methods (Omnidata v1 [11], v2 [22], and the ultra-recent DSINE [3]) as the baselines.”
UniDepth: Universal Monocular Metric Depth Estimation (2024, methodology): “On the other hand, scale-agnostic depth methods, including MiDaS [35], Om-niData [12], and LeReS [47], show robust generalization by training on extensive datasets.”
SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion (2024, methodology): “We further employ a normal loss using the estimated mono normal by Omnidata [11], which is defined as the cosine similarity between the rendered normal n and estimated pseudo ground truths ¯ n : L normal = 1 − ( n · ¯ n ) .”
Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Image (2023, background): “[13] also merged millions of data to build a strong depth prediction model.”
OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generation (2023, background): “MonoSDF is enhanced by depth and normal estimations from pre-trained networks [22], and it claims a superior performance on DTU with only 3 views as input.”
MultiMAE: Multi-modal Multi-task Masked Autoencoders (2022, methodology): “To pseudo label depth, we use a DPT-Hybrid [66] that was trained on Omnidata [27].”
Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation (2023, unknown): “The step from CNNs to vision transformers has further boosted performance, as evidenced by DPT [36] and Omnidata [13].”
MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction (2022, methodology): “Recent work [16, 52, 53], in particular Omnidata [16], has made significant headway in terms of prediction quality and generalization to new scenes using very large datasets for training.”
Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers (2023, unknown): “Furthermore, these methods require a time-intensive, object-specific optimization process to distill diffusion priors [7, 33, 40, 50, 60], combined with monocular cues (depth or normals [14, 38]), into neural 3D representations [43, 58].”
Metric3D v2: A Versatile Monocular Geometric Foundation Model for Zero-Shot Metric Depth and Surface Normal Estimation (2024, unknown): “[33] and ‘OmniData’[35]).”
MoGe: Unlocking Accurate Monocular Geometry Estimation for Open-Domain Images with Optimal Training Supervision (2024, unknown): “3.”