Flash Photography for Data-Driven Hidden Scene Recovery
By Matthew Tancik, Guy Satat, Ramesh Raskar. 2018
38 citations recorded by the source corpus.
Vehicles, search and rescue personnel, and endoscopes use flash lights to locate, identify, and view objects in their surroundings. Here we show the first steps of how all these tasks can be done around corners with consumer cameras. Recent techniques for NLOS imaging using consumer cameras have not been able to both localize and identify the hidden object. We introduce a method that couples traditional geometric understanding and data-driven techniques. To avoid the limitation of large dataset gathering, we train the data-driven models on rendered samples to computationally recover the hidden scene on real data. The method has three independent operating modes: 1) a regression output to localize a hidden object in 2D, 2) an identification output to identify the object type or pose, and 3) a generative network to reconstruct the hidden scene from a new viewpoint. The method is able to localize 12cm wide hidden objects in 2D with 1.7cm accuracy. The method also identifies the hidden object class with 87.7% accuracy (compared to 33.3% random accuracy). This paper also provides an analysis on the distribution of information that encodes the occluded object in the accessible scene. We show that, unlike previously thought, the area that extends beyond the corner is essential for accurate object localization and identification.
Subfields: Computer Vision and Pattern Recognition, Environmental Engineering, Instrumentation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Ramesh Raskar
- Praneeth Vepakomma
- Tristan Swedish
- Vivek Sharma
- Jayashree Kalpathy-Cramer
- Kenglun Chang
- Luc Van Gool
In Environmental Engineering
- Jayashree Kalpathy-Cramer
- Kenglun Chang
- Tristan Swedish
- Praneeth Vepakomma
- Vivek Sharma
- Ramesh Raskar
In Instrumentation
- Jayashree Kalpathy-Cramer
- Kenglun Chang
- Tristan Swedish
- Praneeth Vepakomma
- Vivek Sharma
- Ramesh Raskar
Citations in the corpus
- ExpertMatcher: Automating ML Model Selection for Clients using Hidden Representations (2019, methodology): “…our proposed method on six datasets, such as objects, text, digits, biological and sensor, namely STl-10 (Coates et al. , 2011), MNIST (LeCun et al. , 1998), HAR (Anguita et al. , 2013), Reuters (Lewis et al. , 2004), Non Line of Sight (Tancik et al. , 2018) and Diabetic Retinopathy (Graham, 2015).”
- ExpertMatcher: Automating ML Model Selection for Users in Resource Constrained Countries (2019, methodology): “Our proposed approach is evaluated on 6 benchmark datasets: STl-10 (Coates et al. , 2011), MNIST (LeCun et al. , 1998), HAR (Anguita et al. , 2013), Reuters (Lewis et al. , 2004), Non Line of Sight (Tancik et al. , 2018) and Diabetic Retinopathy (Graham, 2015).”