Multi-view 3D Object Detection Network for Autonomous Driving
By Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, Tian Xia. 2016
3,211 citations recorded by the source corpus.
This paper aims at high-accuracy 3D object detection in autonomous driving scenario. We propose Multi-View 3D networks (MV3D), a sensory-fusion framework that takes both LIDAR point cloud and RGB images as input and predicts oriented 3D bounding boxes. We encode the sparse 3D point cloud with a compact multi-view representation. The network is composed of two subnetworks: one for 3D object proposal generation and another for multi-view feature fusion. The proposal network generates 3D candidate boxes efficiently from the birds eye view representation of 3D point cloud. We design a deep fusion scheme to combine region-wise features from multiple views and enable interactions between intermediate layers of different paths. Experiments on the challenging KITTI benchmark show that our approach outperforms the state-of-the-art by around 25% and 30% AP on the tasks of 3D localization and 3D detection. In addition, for 2D detection, our approach obtains 14.9% higher AP than the state-of-the-art on the hard data among the LIDAR-based methods.
Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, 3D Vision & Pose, Robotics & Control
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Aerospace Engineering
- Hongsheng Li
- Shaoshuai Shi
- Xiaogang Wang
- Philipp Krahenbuhl
- Jiaya Jia
- Andreas Geiger
- Jianping Shi
- Li Jiang
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Xiaogang Wang
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Jitendra Malik
- Alan L. Yuille
- Bernt Schiele
In 3D Vision & Pose
- Andreas Geiger
- Raquel Urtasun
- Leonidas J. Guibas
- Jiaya Jia
- Charles Ruizhongtai Qi
- Xiaogang Wang
- Hongsheng Li
- Shaoshuai Shi
In Robotics & Control
- Raquel Urtasun
- Yuke Zhu
- Silvio Savarese
- Li Fei-Fei
- Roberto Mart'in-Mart'in
- Andreas Geiger
- Lei Tai
- Binh Yang
Citations in the corpus
- VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection (2017, methodology): “Here, we aim to directly estimate the oriented 3D box and normalize Δx and Δy homogeneously with the diagonal d, which is different from [34, 40, 22, 21, 4, 3, 5].”
- A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, positive): “From image classification (Krizhevsky et al., 2012; He et al., 2016; Huang et al., 2017), to semantic segmentation (Long et al., 2015; Chen et al., 2016), from playing Atari games at the human-level with only pixel inputs (Mnih et al., 2015, 2016), to learning policies capable of driving real…”
- Static-Dynamic Co-Teaching for Class-Incremental 3D Object Detection (2021, background): “Many deep learning-based approaches (Li, Zhang, and Xia 2016; Chen et al. 2017; Beltrán et al. 2018; Yan, Mao, and Li 2018; Yang, Luo, and Urtasun 2018; Zeng et al. 2018; Zhou and Tuzel 2018; Chen et al. 2019; Lang et al. 2019; Qi et al. 2019; Shi, Wang, and Li 2019; Yang et al. 2019; Zhou et al.…”
- 3D fully convolutional network for vehicle detection in point cloud (2016, methodology): “For example, [6] uses bird eye view and the front view projection of point cloud as input data.”
- Towards Safe Autonomous Driving: Capture Uncertainty in the Deep Neural Network For Lidar 3D Vehicle Detection (2018, background): “Therefore, it is natural to combine these two sensors [11]–[17].”
- A Survey of Autonomous Driving: Common Practices and Emerging Technologies (2019, methodology): “The MV3D algorithm [169] used camera images, depth images, as well as multi-channel BV images; each channel corresponding to a different range of heights, so as to minimize these occlusions.”
- Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerialization (2019, background): “Deep learning is rapidly pushing the limits of memory capacity on neural network accelerators as researchers train neural networks on high-resolution images [15, 32, 55], 3D point-clouds [10, 59], and long Natural Language Processing (NLP) sequence data [11, 14, 56].”
- nuScenes: A Multimodal Dataset for Autonomous Driving (2019, background): “And while there are several works that have proposed fusion methods based on cameras and lidar [48, 14, 64, 52, 81, 75, 29], PointPillars [51] showed a lidar-only method that performed on par with existing fusion based methods.”
- Orientation-boosted Voxel Nets for 3D Object Recognition (2016, neutral): “[5] mix both 2D and 3D data for detection.”
- PIXOR: Real-time 3D Object Detection from Point Clouds (2018, methodology): “We compare with 3D object detectors that use LIDAR on KITTI benchmark: VeloFCN [20], 3D FCN [19] and MV3D [3].”
- MVX-Net: Multimodal VoxelNet for 3D Object Detection (2019, methodology): “performance than [5] especially in the small object category by designing a more advanced RPN that employs highresolution feature maps.”
- A Survey of the Four Pillars for Small Object Detection: Multiscale Representation, Contextual Information, Super-Resolution, and Region Proposal (2022, background): “As a promising technology related to computer vision, object detection has been applied to many application scenes, such as pedestrian detection [1], [2], face detection [3]–[5], autonomous driving [6]–[8], and robotic vision [9]–[12].”
- TANet: Robust 3D Object Detection from Point Clouds with Triple Attention (2019, methodology): “…two categories: the front view based approaches (Song and Chandraker 2015; Chen et al. 2016; Mousavian et al. 2017), and the bird’s eye view based approaches (Chen et al. 2017; Ku et al. 2018; Yang, Luo, and Urtasun 2018; Simon et al. 2018; Li, Zhang, and Xia 2016; Yang, Liang, and Urtasun 2018).”
- PVN3D: A Deep Point-Wise 3D Keypoints Voting Network for 6DoF Pose Estimation (2019, background): “It is an important component in many real-world applications, such as robotic grasping and manipulation [6, 48, 55], autonomous driving [11, 5, 53], augmented reality [31] and so on.”
- Lidar for Autonomous Driving: The Principles, Challenges, and Trends for Automotive Lidar and Perception Systems (2020, methodology): “A more complex neural network was proposed in [48], where CNNs are utilized in both a range image and BEV of lidar data and then fused with camera detections.”
- Class-balanced Grouping and Sampling for Point Cloud 3D Object Detection (2019, methodology): “Several works [3, 13, 15, 29, 14] convert point cloud into bird-view format and apply 2D CNN to get 3D object detection results.”
- A Survey of Deep Learning-Based Object Detection (2019, methodology): “[296] take both LiDAR point cloud and RGB images as input then predict oriented 3D bounding boxes for high-accuracy 3D object detection.”
- Improving 3D Object Detection with Channel-wise Transformer (2021, methodology): “Pioneer work [3] encoded the point clouds as 2D bird-view feature maps to generate highly accurate 3D candidate boxes, motivating many efficient bird-view representation-based methods.”
- IntentNet: Learning to Predict Intention from Raw Sensor Data (2018, methodology): “Approaches that use the projection representation (VeloFCN [15], MV3D [16]) or handle point clouds directly (PointNet [17]) have also been proposed.”