ImVoteNet: Boosting 3D Object Detection in Point Clouds With Image Votes

By Charles Ruizhongtai Qi, Xinlei Chen, Or Litany, Leonidas J. Guibas. Computer Vision and Pattern Recognition, 2020

300 citations recorded by the source corpus.

3D object detection has seen quick progress thanks to advances in deep learning on point clouds. A few recent works have even shown state-of-the-art performance with just point clouds input (e.g. VoteNet). However, point cloud data have inherent limitations. They are sparse, lack color information and often suffer from sensor noise. Images, on the other hand, have high resolution and rich texture. Thus they can complement the 3D geometry provided by point clouds. Yet how to effectively use image information to assist point cloud based detection is still an open question. In this work, we build on top of VoteNet and propose a 3D detection architecture called ImVoteNet specialized for RGB-D scenes. ImVoteNet is based on fusing 2D votes in images and 3D votes in point clouds. Compared to prior work on multi-modal detection, we explicitly extract both geometric and semantic features from the 2D images. We leverage camera parameters to lift these features to 3D. To improve the synergy of 2D-3D feature fusion, we also propose a multi-tower training scheme. We validate our model on the challenging SUN RGB-D dataset, advancing state-of-the-art results by 5.7 mAP. We also provide rich ablation studies to analyze the contribution of each design choice.

Subfields: Computer Vision and Pattern Recognition, Geology

View Paper: https://www.semanticscholar.org/paper/ddb42aace56362ea9725a33815a64bb213d0329a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Geology

Citations in the corpus

Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution (2020, background): “There are also 3D models tailored for specific tasks such as detection [38, 39, 41, 47–49, 70, 73] and instance segmentation [16, 21, 23, 71] built upon these modules.”
MoVie: Revisiting Modulated Convolutions for Visual Counting and Beyond (2021, background): “3 (b)) likely suffers from feature co-adaptation (Hinton et al., 2012), we follow a three-branch training scheme (Wang et al., 2019; Qi et al., 2020) by adding two auxiliary , training-only branches: One trains the original VQA model just with i , and the other trains a normal MoVie with v and MLP.”
Deep Learning for 3D Point Clouds: A Survey (2019, methodology): “[150] proposed an ImVoteNet detector by fusing 2D object detection cues (e.”
PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding (2020, background): “Among many different choices [79, 29, 46, 44], we identify the recently proposed VoteNet [45] as a good candidate for three main reasons.”
Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contexts (2020, unknown): “A variety of datasets [2, 13, 54, 18, 56] and algorithms have been proposed for 3D object detection [45, 44, 43, 24], semantic segmentation [46, 57, 63, 58, 20, 12] and instance segmentation [60, 30, 71, 36, 61, 69, 31, 15, 34, 33].”
TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers (2022, background): “LiDAR-only 3D Detection aims to predict 3D bounding boxes of objects in given point clouds [3, 4, 27, 28, 38, 46, 53, 66, 68, 69].”
What Makes Training Multi-Modal Classification Networks Hard? (2019, methodology): “We note that G-Blend is task-agnostic, architecture-agnostic and applicable to other scenarios (e.g. used in [39] to combine point cloud with RGB for 3D object detection)”
Group-Free 3D Object Detection via Transformers (2021, methodology): “0 imVoteNet [25]∗ PointNet++ point+RGB 63.”
Vision-based robotic grasping from object localization, object pose estimation to grasp estimation for parallel grippers: a review (2020, unknown): “, 2020b], ImVoteNet [Qi et al., 2020] VoxelNet [Zhou and Tuzel, 2018], SECOND [Yan et al.”
ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data (2021, background): “A variety of methods are addressing different parts of the challenge, like depth estimation, 3D reconstruction, instance segmentation [1, 2, 3], object detection [4, 5, 6, 7] and more.”
Open-vocabulary Queryable Scene Representations for Real World Planning (2022, background): “Many representations are proposed, ranging from a faithful 3D recontruction [12] of the environment, to more object-centric ones [13], [14], such as object detection bounding boxes [15] and 3D bounding boxes [16].”
Unseen Object Instance Segmentation for Robotic Environments (2020, methodology): “A few of these methods embrace center voting-based techniques [55], [56], [57], [58], [13].”