Learning the Depths of Moving People by Watching Frozen People

By Zhengqi Li, Tali Dekel, Forrester Cole, Richard Tucker, Noah Snavely, Ce Liu, William T. Freeman. 2019

275 citations recorded by the source corpus.

We present a method for predicting dense depth in scenarios where both a monocular camera and people in the scene are freely moving. Existing methods for recovering depth for dynamic, non-rigid objects from monocular video impose strong assumptions on the objects’ motion and may only recover sparse depth. In this paper, we take a data-driven approach and learn human depth priors from a new source of data: thousands of Internet videos of people imitating mannequins, i.e., freezing in diverse, natural poses, while a hand-held camera tours the scene. Since the people are stationary, training data can be created from these videos using multi-view stereo reconstruction. At inference time, our method uses motion parallax cues from the static areas of the scenes, and shows clear improvement over state-of-the-art monocular depth prediction methods. We demonstrate our method on real-world sequences of complex human actions captured by a moving hand-held camera, and show various 3D effects produced using our predicted depth.

Subfields: Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer (2019, methodology): “For the TUM dataset [37], we use the dynamic subset that features humans in indoor environments [38].”
PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking (2023, methodology): “Finally, we import camera trajectories computed from real video [35], and attach additional cameras to the synthetic humans’ heads, giving challenging multi-view data of the scenes.”
CameraCtrl: Enabling Camera Control for Text-to-Video Generation (2024, unknown): “Besides, there are some other datasets with characteristics similar to RealEstate10K, such as ACID (Liu et al., 2021) and MannequinChallenge (Li et al., 2019), but their data volume is much smaller than RealEstate10K.”
Sim2real transfer learning for 3D human pose estimation: motion to the rescue (2019, background): “Among these, we are particularly related to recent works which leverage temporal cues from videos to gain extra information about depth [6, 14, 27, 28, 35, 41, 58, 90, 91].”
Pix2Shape: Towards Unsupervised Learning of 3D Scenes from Images Using a View-Based Representation (2020, unknown): “Recent approaches rely on using other forms of 3D supervision, such asmultiple views of the same object (Yan et al. 2016; Tulsiani et al. 2017; Li et al. 2019), 2.5D supervision (Wu et al. 2016a, 2017), key-point (Kar et al. 2014; Novotný et al. 2019) and silhouette annotations (Wiles and Zisserman…”
Self-Supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidance (2020, background): “Detecting Non-Moving DC Objects: Inspired by [33], we do not want to exclude the DC objects completely, instead we only learn from them, when they are not in motion.”
Single-View View Synthesis With Multiplane Images (2020, methodology): “This idea has been used in classical computer vision, notably in stereo correspondence [25], and also in a variety of different smoothness losses for learning depth prediction [9, 17, 33].”
Monocular human pose estimation: A survey of deep learning-based methods (2020, background): “Deep neural networks have the capability to estimate the dense depth (Li et al., 2015a, 2018a, 2019) and sparse depth points (joints) as well from monocular images.”
Consistent video depth estimation (2020, methodology): “Learning-based methods (e.g., [Li et al. 2019]) predict dense depth for each frame but the video reconstruction is flickering and geometrically inconsistent.”
Dual Attention Guided Gaze Target Detection in the Wild (2021, background): “In order to investigate whether our model is robust to different priori depth maps, we present a comparative study using four recent monocular depth estimation meth-ods [16, 3, 27, 35].”
NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, background): “a pretrained (fine-tuned on the scene) monocular depth network [88] to extract a depth map prior.”
TAP-Vid: A Benchmark for Tracking Any Point in a Video (2022, unknown): “This work is thus often restricted to rigid scenes [45, 87], and is benchmarked accordingly [3, 10, 36, 36, 37, 57, 88].”
Defocus Blur Detection via Depth Distillation (2020, background): “[20] generate the multi-view disparity of humans from video of people who are frozen in place, and this task only works when the person are in the scene.”
Watching the World Go By: Representation Learning from Unlabeled Videos (2020, background): “There is a long history of using videos for low level [20, 23, 32] and high-level tasks [26, 36].”
TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement (2023, background): “This line work is thus often restricted to rigid scenes [37, 70], and is benchmarked accordingly [1, 10, 27, 28, 48, 71].”