FlowNet3D: Learning Scene Flow in 3D Point Clouds

By Xingyu Liu, Charles Ruizhongtai Qi, Leonidas J. Guibas. 2018

564 citations recorded by the source corpus.

Many applications in robotics and human-computer interaction can benefit from understanding 3D motion of points in a dynamic environment, widely noted as scene flow. While most previous methods focus on stereo and RGB-D images as input, few try to estimate scene flow directly from point clouds. In this work, we propose a novel deep neural network named FlowNet3D that learns scene flow from point clouds in an end-to-end fashion. Our network simultaneously learns deep hierarchical features of point clouds and flow embeddings that represent point motions, supported by two newly proposed learning layers for point sets. We evaluate the network on both challenging synthetic data from FlyingThings3D and real Lidar scans from KITTI. Trained on synthetic data only, our network successfully generalizes to real scans, outperforming various baselines and showing competitive results to the prior art. We also demonstrate two applications of our scene flow output (scan registration and motion segmentation) to show its potential wide use cases.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

1. Yulan Guo
2. Song Han
3. Zhijian Liu
4. Yujun Lin
5. Qingyong Hu
6. Mohammed Bennamoun
7. Leon Klein
8. Yifan Wang

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Xiaogang Wang
4. Song Han
5. Jie Zhou
6. Jiwen Lu
7. Yi Yang
8. Ming-Hsuan Yang

Citations in the corpus

• Equivariant Flows: exact likelihood generative learning for symmetric densities (2020, methodology): “Symmetries in flow models have been discussed in the context of permutations in graphs (Liu et al., 2019a).”
• Argoverse: 3D Tracking and Forecasting With Rich Maps (2019, background): “Recent work demonstrates how 3D instance segmentation and 3D motion (in the form of 3D scene flow, or per-point velocity vectors) can be estimated directly on point cloud input with deep networks [37, 23].”
• Deep Learning for 3D Point Clouds: A Survey (2019, background): “Liu et al. [142] proposed FlowNet3D to directly learn scene flows from a pair of consecutive point clouds.”
• Point-Voxel CNN for Efficient 3D Deep Learning (2019, methodology): “On the same dataset for 3D object detection, FlowNet3D [26] extends PointNet++ [34] for scene flow estimation.”
• Back to Reality: Weakly-supervised 3D Object Detection with Shape-guided Label Enhancement (2022, methodology): “Existing approaches can be divided into two categories: supervised [2, 3, 6, 40] and unsupervised [8, 19, 24, 30, 42].”