DepthSynth: Real-Time Realistic Synthetic Data Generation from CAD Models for 2.5D Recognition

By Benjamin Planche, Ziyan Wu, Kai Ma, Shanhui Sun, Stefan Kluckner, Terrence Chen, A. Hutter, Sergey Zakharov, H. Kosch, Jan Ernst. 2017

73 citations recorded by the source corpus.

Recent progress in computer vision has been dominated by deep neural networks trained over larges amount of labeled data. Collecting such datasets is however a tedious, often impossible task; hence a surge in approaches relying solely on synthetic data for their training. For depth images however, discrepancies with real scans still noticeably affect the end performance. We thus propose an end-to-end framework which simulates the whole mechanism of these devices, generating realistic depth data from 3D models by comprehensively modeling vital factors e.g. sensor noise, material reflectance, surface geometry. Not only does our solution cover a wider range of sensors and achieve more realistic results than previous methods, assessed through extended evaluation, but we go further by measuring the impact on the training of neural networks for various recognition tasks; demonstrating how our pipeline seamlessly integrates such architectures and consistently enhances their performance.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Geology, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

1. Pieter Abbeel
2. Wojciech Zaremba
3. Wadim Kehl
4. Sergey Levine
5. Sergey Zakharov
6. Joshua Tobin
7. Slobodan Ilic
8. Sergey I. Nikolenko

In Computer Vision and Pattern Recognition

1. Sergey I. Nikolenko
2. Pieter Abbeel
3. Luc Van Gool
4. Trevor Darrell
5. Jitendra Malik
6. Dacheng Tao
7. Sergey Levine
8. Kristen Grauman

In Geology

1. Pieter Abbeel
2. Sergey I. Nikolenko
3. Slobodan Ilic
4. Vladlen Koltun

In Evaluation & Benchmarks

1. Pieter Abbeel
2. Wojciech Zaremba
3. Sergey I. Nikolenko
4. Richard J. Chen
5. Wadim Kehl
6. Slobodan Ilic
7. Sergey Levine
8. David R Ha

Citations in the corpus

• Domain randomization for transferring deep neural networks from simulation to the real world (2017, background): “Though using realistic RGB rendering alone has had limited success for transferring to real robotic tasks [16], incorporating realistic simulation of depth information can allow models trained on rendered images to transfer reasonably well to the real world [32].”
• DeceptionNet: Network-Driven Domain Randomization (2019, background): “Producing synthetic depth images is rather easy and simulation of their sensor noise [16, 32] has seen great advances.”
• Synthetic Data for Deep Learning (2019, background): “[455] consider the problem of more realistic simulation of depth data from real sensors and present DepthSynth, an end-to-end framework able to generate realistic depth data rather than purely synthetic perfect depth maps; they show that this added realism leads to improvements with modern 2.”
• Unsupervised Reverse Domain Adaptation for Synthetic Medical Images via Adversarial Training (2017, background): “Despite the widespread use of synthetic data for training deep networks for real world images [29, 8, 31, 20], its use for medical imaging applications has been relatively limited.”