Frustum ConvNet: Sliding Frustums to Aggregate Local Point-Wise Features for Amodal

By Zhixin Wang, K. Jia. IEEE/RJS International Conference on Intelligent RObots and Systems, 2019

501 citations recorded by the source corpus.

In this work, we propose a novel method termed Frustum ConvNet (F-ConvNet) for amodal 3D object detection from point clouds. Given 2D region proposals in an RGB image, our method first generates a sequence of frustums for each region proposal, and uses the obtained frustums to group local points. F-ConvNet aggregates point-wise features as frustum-level feature vectors, and arrays these feature vectors as a feature map for use of its subsequent component of fully convolutional network (FCN), which spatially fuses frustum-level features and supports an end-to-end and continuous estimation of oriented boxes in the 3D space. We also propose component variants of F-ConvNet, including an FCN variant that extracts multi-resolution frustum features, and a refined use of F-ConvNet over a reduced 3D space. Careful ablation studies verify the efficacy of these component variants. F-ConvNet assumes no prior knowledge of the working 3D environment and is thus dataset-agnostic. We present experiments on both the indoor SUN-RGBD and outdoor KITTI datasets. F-ConvNet outperforms all existing methods on SUN-RGBD, and at the time of submission it outperforms all published works on the KITTI benchmark. Code has been made available at: https://github.com/zhixinwang/frustum-convnet.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Geology

View Paper: https://www.semanticscholar.org/paper/db462a8de0449fa996ff32f415c688e2d26df5ae

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Geology

Citations in the corpus

Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision (2019, background): “Current applications include many safety-critical tasks, such as street-scene semantic segmentation [10, 50, 6, 52], automotive 3D object detection [48, 44, 29, 42, 55] and depth completion [46, 33, 12].”
PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detection (2021, methodology): “In contrast, the point-based methods (Qi et al., 2018; Shi et al., 2019; Wang and Jia, 2019) naturally preserve accurate point locations in feature extraction but are generally computationally-intensive on handling large-scale points.”
Disentangling Monocular 3D Object Detection (2019, methodology): “Even though there are promising methods based on multi-sensor fusion (usually exploiting LIDAR information [14, 33, 30] next to RGB images), 3D detection results produced from a single, monocular RGB input image lag considerably behind.”
IoU Loss for 2D/3D Object Detection (2019, background): “We have listed nearly all the top results with publications here including: multi-modalities fusionbased [24, 8, 25, 26, 27, 28], one-stage- [10, 9, 11] and twostage-based [12] approaches.”
SMOKE: Single-Stage Monocular 3D Object Detection via Keypoint Estimation (2020, background): “LiDAR-image fusion methods [17, 33, 5] learn relevant features from both the point clouds and the images together.”
3DSSD: Point-Based 3D Single Stage Object Detector (2020, methodology): “Moreover, we prove its superiority by comparing with multi-sensors methods of MMF [14] and F-ConvNet [27] – our method intriguingly achieves 2.14% and 3.18% improvement respectively.”
CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object Detection (2020, background): “Frustum PointNet [24], Pointfusion [13] and Frustum ConvNet [25] are the representatives of 2D driven 3D detectors, which exploit mature 2D detectors to generate 2D proposals and narrow down the 3D processing domain to the corresponding cropped region in the image.”
ST3D: Self-training for Unsupervised Domain Adaptation on 3D Object Detection (2021, methodology): “There are also some other methods [28, 42] that utilize 2D images for generating 2D box proposals which are further employed to crop the object-level point clouds for generating 3D bounding boxes.”
Multimodal Virtual Point 3D Detection (2021, background): “Frustum PointNet [38] and Frustum ConvNet [58] ﬁrst detect objects in image space to identify regions of interest in the point cloud for further processing.”
Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Review (2020, unknown): “Frustum based [57] [65] [58] [59] [61] [63] [62] 1.”
FVNet: 3D Front-View Proposal Generation for Real-Time Object Detection from Point Clouds (2019, methodology): “F-Pointnet [18] and FConvNet [29] directly operated on raw point clouds by popping up RGB-D scans, without any loss of data, leading to precise detection.”
Deep Learning for 3D Point Clouds: A Survey (2019, methodology): “[144] generated a sequence of frustums along the frustum axis for each 2D region and applied PointNet [5] to extract features for each frustum.”
BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation (2022, background): “[45], [46], [47] all lift image proposals into a 3D frustum.”
SE-SSD: Self-Ensembling Single-Stage Object Detector From Point Cloud (2021, unknown): “41 100 F-ConvNet [28] IROS 2019 LiDAR+RGB 87.”
3D-CVF: Generating Joint Camera and LiDAR Features Using Cross-View Spatial Feature Fusion for 3D Object Detection (2020, methodology): “Recently, these studies have been extended to the 3D object detection task [28,2,7,23,3,16,30,27,11,8,12,29,25], where the locations of the objects should be identiﬁed in 3D world coordinates.”
Learning Depth-Guided Convolutions for Monocular 3D Object Detection (2019, background): “[49] aggregates point-wise features as frustum-level feature vectors.”
Structure Aware Single-Stage 3D Object Detection From Point Cloud (2020, methodology): “Some image-driven methods [16, 23] have been proposed to lift a set of 3D regions of interest (RoI) from the image and then apply a PointNet to extract RoI feature by gathering the interior points with transformed canonical coordinates.”
From Points to Parts: 3D Object Detection From Point Cloud With Part-Aware and Part-Aggregation Network (2019, methodology): “Compared with the previous 3D object detection methods [1], [4], [5], [6], [7], [26], [29], [64], (1) our proposed method for the ﬁrst time introduces the learning of intra-object part locations to improve the performance of 3D object detection from point cloud, where the predicted intra-object…”
DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection (2022, background): “Some approaches [24, 39] first detect objects in 2D images then use the information to further process the point cloud.”