Joint 3D Proposal Generation and Object Detection from View Aggregation

By Jason Ku, Melissa Mozifian, Ali Harakeh, Steven L. Waslander. 2017

1,597 citations recorded by the source corpus.

We present AVOD, an Aggregate View Object Detection network for autonomous driving scenarios. The proposed neural network architecture uses LIDAR point clouds and RGB images to generate features that are shared by two subnetworks: a region proposal network (RPN) and a second stage detector network. The proposed RPN uses a novel architecture capable of performing multimodal feature fusion on high resolution feature maps to generate reliable 3D object proposals for multiple object classes in road scenes. Using these proposals, the second stage detection network performs accurate oriented 3D bounding box regression and category classification to predict the extents, orientation, and classification of objects in 3D space. Our proposed architecture is shown to produce state of the art results on the KITTI 3D object detection benchmark [1] while running in real time with a low memory footprint, making it a suitable candidate for deployment on autonomous vehicles. Code is available at

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Environmental Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Environmental Engineering

Citations in the corpus

nuScenes: A Multimodal Dataset for Autonomous Driving (2019, background): “And while there are several works that have proposed fusion methods based on cameras and lidar [48, 14, 64, 52, 81, 75, 29], PointPillars [51] showed a lidar-only method that performed on par with existing fusion based methods.”
Multi-Task Multi-Sensor Fusion for 3D Object Detection (2019, background): “Recently, several attempts [5, 17, 12, 13] have been developed to fuse information from multiple sensors.”
SE-SSD: Self-Ensembling Single-Stage Object Detector From Point Cloud (2021, methodology): “54 170 AVOD [10] IROS 2018 LiDAR+RGB 83.”
TANet: Robust 3D Object Detection from Point Clouds with Triple Attention (2019, background): “…two categories: the front view based approaches (Song and Chandraker 2015; Chen et al. 2016; Mousavian et al. 2017), and the bird’s eye view based approaches (Chen et al. 2017; Ku et al. 2018; Yang, Luo, and Urtasun 2018; Simon et al. 2018; Li, Zhang, and Xia 2016; Yang, Liang, and Urtasun 2018).”
FUTR3D: A Unified Sensor Fusion Framework for 3D Detection (2022, methodology): “Proposalbased methods either propose bounding boxes from the LiDAR point clouds and then extract corresponding features from camera images [6, 11], or propose frustums from the images and further refine bounding boxes according to point clouds in the frustums [21].”
Not All Points Are Equal: Learning Highly Efficient Point-based Detectors for 3D LiDAR Point Clouds (2022, neutral): “Due to the unstructured and orderless nature of 3D point clouds, early works usually first convert the raw point clouds into intermediate regular representation, including projecting the 3D point clouds into 2D images from birdseye-view or frontal view [1, 17, 18, 41, 42, 50, 60], or transformed int”
PointRCNN: 3D Object Proposal Generation and Detection From Point Cloud (2018, methodology): “[4, 14, 42, 17, 41] projected point cloud to bird’s view and utilized 2D CNN to learn the point cloud features for 3D box generation.”
Deep Continuous Fusion for Multi-sensor 3D Object Detection (2018, methodology): “Other approaches [6,18,8] apply 2D convolutional networks on both camera image and LIDAR bird’s eye view (BEV) representations, and fuse them at the intermediate region-wise convolutional feature map via feature concatenation.”
Unifying Voxel-based Representation with Transformer for 3D Object Detection (2022, background): “And instance-level fusion [33, 34, 35] is usually applied at the later stage to combine object-level features.”
3DSSD: Point-Based 3D Single Stage Object Detector (2020, methodology): “AVOD [11] extends MV3D by introducing image features in the proposal generation stage.”
Categorical Depth Distribution Network for Monocular 3D Object Detection (2021, background): “LiDAR [21, 50, 51] and stereo [46, 45, 28, 11] sensors have a long history of use for 3D perception tasks, showing excellent results on 3D object detection benchmarks such as the KITTI 3D object detection benchmark [16] due to their ability to generate precise 3D measurements.”
YOLO3D: End-to-end real-time 3D Oriented Object Bounding Box Detection from LiDAR Point Cloud (2018, methodology): “AVOD[13] converts the orientation vector to sine and cosine.”
IoU Loss for 2D/3D Object Detection (2019, background): “We have listed nearly all the top results with publications here including: multi-modalities fusionbased [24, 8, 25, 26, 27, 28], one-stage- [10, 9, 11] and twostage-based [12] approaches.”
PointPainting: Sequential Fusion for 3D Object Detection (2019, background): “Object-centric fusion, pioneered by MV3D [3] and AVOD [9], is the most obvious choice for a two-stage architecture.”
Seeing Through Fog Without Seeing Fog: Deep Multimodal Sensor Fusion in Unseen Adverse Weather (2019, methodology): “As large automotive datasets [66, 19, 59] cover limited sensory inputs, existing fusion methods have been proposed mostly for lidar-camera setups [65, 56, 11, 36, 43].”
ST3D: Self-training for Unsupervised Domain Adaptation on 3D Object Detection (2021, background): “Some previous work [6, 23, 47] directly projects the irregular point clouds to 2D bird-view maps such that the task could be resolved by previous 2D detection methods.”
A Survey on 3D Object Detection Methods for Autonomous Driving Applications (2019, methodology): “Two of these methods [6], [64] use 3D region proposal networks (RPNs) to generate 3D Regions of Interest (ROI) which are then projected to the specific views and used to predict classes and 3D bounding boxes.”
BEVFormer v2: Adapting Modern Image Backbones to Bird's-Eye-View Recognition via Perspective Supervision (2022, background): “Although two-stage detectors are common in LiDARbased 3D object detection [1, 5, 12, 16, 29, 39, 42], their application in camera-based 3D detection is far less well known.”