Frustum PointNets for 3D Object Detection from RGB-D Data
By Charles Ruizhongtai Qi, Wei David Liu, Chenxia Wu, Hao Su, Leonidas J. Guibas. 2017
2,541 citations recorded by the source corpus.
In this work, we study 3D object detection from RGBD data in both indoor and outdoor scenes. While previous methods focus on images or 3D voxels, often obscuring natural 3D patterns and invariances of 3D data, we directly operate on raw point clouds by popping up RGB-D scans. However, a key challenge of this approach is how to efficiently localize objects in point clouds of large-scale scenes (region proposal). Instead of solely relying on 3D proposals, our method leverages both mature 2D object detectors and advanced 3D deep learning for object localization, achieving efficiency as well as high recall for even small objects. Benefited from learning directly in raw point clouds, our method is also able to precisely estimate 3D bounding boxes even under strong occlusion or with very sparse points. Evaluated on KITTI and SUN RGB-D 3D detection benchmarks, our method outperforms the state of the art by remarkable margins while having real-time capability.
Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Geology
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Aerospace Engineering
- Shaoshuai Shi
- Hongsheng Li
- Xiaogang Wang
- Jiaya Jia
- Philipp Krahenbuhl
- Li Jiang
- Leonidas J. Guibas
- Jianping Shi
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Xiaogang Wang
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Jitendra Malik
- Alan L. Yuille
- Andrea Vedaldi
In Geology
- Or Litany
- Matthias Nießner
- Charles Ruizhongtai Qi
- Shaoshuai Shi
- Leonidas J. Guibas
- Li Jiang
- Hongsheng Li
- Jiaya Jia
Citations in the corpus
- Dynamic Graph CNN for Learning on Point Clouds (2018, background): “rations or instability of these techniques in the presence of noise. A few of the many recent applications of point cloud processing and analysis include indoor navigation [57], self-driving vehicles [33], robotics [40], and shape synthesis and modeling [14]. Modern applications demand high-level pr”
- Towards Safe Autonomous Driving: Capture Uncertainty in the Deep Neural Network For Lidar 3D Vehicle Detection (2018, methodology): “[12] use RGB camera images to draw 2D object bounding boxes, which build frustums in the Lidar point cloud.”
- A Survey of Autonomous Driving: Common Practices and Emerging Technologies (2019, neutral): “intcloud data. Algorithm T [s] Easy Moderate Hard PointRCNN [179] 0.10 85.9 75.8 68.3 PointPillars [180] 0.02 79.1 75.0 68.3 SECOND [140] 0.04 83.1 73.7 66.2 IPOD [181] 0.20 82.1 72.6 66.3 F-PointNet [182] 0.17 81.2 70.4 62.2 VoxelNet (Lidar) [167] 0.23 77.5 65.1 57.7 MV3D (Lidar) [172] 0.24 66.8 52”
- nuScenes: A Multimodal Dataset for Autonomous Driving (2019, background): “And while there are several works that have proposed fusion methods based on cameras and lidar [48, 14, 64, 52, 81, 75, 29], PointPillars [51] showed a lidar-only method that performed on par with existing fusion based methods.”
- One Million Scenes for Autonomous Driving: ONCE Dataset (2021, background): “Many techniques have been explored for 3D object de92 tection in driving scenarios, and they can be broadly categorized into two classes: 1) Single-modality 93 3D detectors [48, 36, 23, 35, 51, 57, 34, 49] are designed to detect objects solely from sparse point 94 clouds.”
- StarNet: Targeted Computation for Object Detection in Point Clouds (2019, methodology): “Other works propose bounding boxes through a computationally intensive, learned proposal system operating on paired camera images [48, 31], with the goal of improving predictive performance by leveraging a camera image to seed a proposal system to maximize recall.”
- BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework (2022, methodology): “LiDAR methods initially lie in two categories based on their feature modality: i) point-based methods that directly operate on the raw LiDAR point clouds [36, 35, 34, 40, 55, 21]; and ii) transforming the original point clouds into a Euclidean feature space, such as 3D voxels [61], feature pillar [1”
- Vision-based robotic grasping from object localization, object pose estimation to grasp estimation for parallel grippers: a review (2020, methodology): “There exist various methods of generating 3D object proposals, which can be roughly divided into three kinds, which are frustum-based methods [Qi et al., 2018; Xu et al., 2018; Wang and Jia, 2019], global regressionbased methods [Song and Xiao, 2016; Chen et al.”
- Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Review (2020, methodology): “Multiple 3D cylinder proposals are first generated based on each 2D bbox using the geometric agreement search [60], which results in smaller but more precise frustum proposals then the F-pointNet [57].”
- SECOND: Sparsely Embedded Convolutional Detection (2018, methodology): “31 N/A N/A N/A N/A N/A N/A F-PointNet [11] 0.”
- M3D-RPN: Monocular 3D Region Proposal Network for Object Detection (2019, methodology): “Currently, the foremost methods [12, 24, 31, 35, 41] on 3D detection rely extensively on expensive LiDAR sensors to provide sparse depth data as input.”
- Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloud (2020, methodology): “Our work is a single-stage detection method without the need to develop a second-stage refinement neural networks like those in [4][16][21][11][13].”
- Complex-YOLO: An Euler-Region-Proposal for Real-Time 3D Object Detection on Point Clouds (2018, methodology): “Direct point cloud processing using Multi-Layer-Perceptrons [5] [10] [11] [23] [24] 2.”
- Cooperative Perception for 3D Object Detection in Driving Scenarios Using Infrastructure Sensors (2019, methodology): “Majority of the existing methods rely on data fusion from different sensor modalities to overcome the limitations of single sensor types and to increase detection performance [6]–[8].”
- BEVFormer v2: Adapting Modern Image Backbones to Bird's-Eye-View Recognition via Perspective Supervision (2022, background): “Although two-stage detectors are common in LiDAR-based 3D object detection [1, 5, 12, 16, 28, 38, 41], their application in camera-based 3D detection is far less well known.”
- 3D Object Detection for Autonomous Driving: A Comprehensive Survey (2022, methodology): “, 2021c) is generally more effective than region-level fusion (Qi et al., 2018; Wang & Jia, 2019).”
- Review: deep learning on 3D point clouds (2020, methodology): “FrustumNet [105] proposed regions in 2D and obtain the 3D frustrum of the region from the point cloud and pass it through PointNet to predict 3D bouding box.”
- PVN3D: A Deep Point-Wise 3D Keypoints Voting Network for 6DoF Pose Estimation (2019, methodology): “The extra depth information allows 2D algorithms to be extend into 3D space with better performance, like Point-Fusion [53], Frustum pointnets[39] and VoteNet[38].”