Layered RGBD scene flow estimation

By Deqing Sun, Erik B. Sudderth, Hanspeter Pfister. 2015

73 citations recorded by the source corpus.

As consumer depth sensors become widely available, estimating scene flow from RGBD sequences has received increasing attention. Although the depth information allows the recovery of 3D motion from a single view, it poses new challenges. In particular, depth boundaries are not well-aligned with RGB image edges and therefore not reliable cues to localize 2D motion boundaries. In addition, methods that extend the 2D optical flow formulation to 3D still produce large errors in occlusion regions. To better use depth for occlusion reasoning, we propose a layered RGBD scene flow method that jointly solves for the scene segmentation and the motion. Our key observation is that the noisy depth is sufficient to decide the depth ordering of layers, thereby avoiding a computational bottleneck for RGB layered methods. Furthermore, the depth enables us to estimate a per-layer 3D rigid motion to constrain the motion of each layer. Experimental results on both the Middlebury and real-world sequences demonstrate the effectiveness of the layered approach for RGBD scene flow estimation.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Scene Flow to Action Map: A New Representation for RGB-D Based Action Recognition with Convolutional Neural Networks (2017, methodology): “In contrast to optical flow which provides the projection of the scene motion onto the image plane, scene flow [41, 6, 26, 8, 11, 37, 32] estimates the actual 3D motion field.”
Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes (2020, background): “Additional work has aimed to predict per-pixel scene flows of dynamic scenes from either monocular or RGBD sequences [8, 26, 30, 41, 47, 71].”
FlowNet3D: Learning Scene Flow in 3D Point Clouds (2018, unknown): “With the recent advent of commodity depth sensors, it has become feasible to estimate scene flow from monocular RGB-D images [9], by generalizing variational 2D flow algorithms to 3D [10, 14] and exploiting more geometric cues provided by the depth channel [21, 11, 23].”