3D-RCNN: Instance-Level 3D Object Reconstruction via Render-and-Compare

By Abhijit Kundu, Yin Li, James M. Rehg. 2018

359 citations recorded by the source corpus.

We present a fast inverse-graphics framework for instance-level 3D scene understanding. We train a deep convolutional network that learns to map image regions to the full 3D shape and pose of all object instances in the image. Our method produces a compact 3D representation of the scene, which can be readily used for applications like autonomous driving. Many traditional 2D vision outputs, like instance segmentations and depth-maps, can be obtained by simply rendering our output 3D scene model. We exploit class-specific shape priors by learning a low dimensional shape-space from collections of CAD models. We present novel representations of shape and pose, that strive towards better 3D equivariance and generalization. In order to exploit rich supervisory signals in the form of 2D annotations like segmentation, we propose a differentiable Render-and-Compare loss that allows 3D shape and pose to be learned with 2D supervision. We evaluate our method on the challenging real-world datasets of Pascal3D+ and KITTI, where we achieve state-of-the-art results.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Objects as Points (2019, methodology): “Fast-RCNN [18] crops image features instead, to save computation.”
Multiview Compressive Coding for 3D Reconstruction (2023, background): “Reminiscent of generalized cylinders [40], some introduce object-specific priors via category-specific 3D templates [25,29,31], pose [42] or symmetries [73].”
Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, background): “One highly related direction is to model and estimate 3D hand joints [5,44,45,53,60,63,66,72,78,79,80,85,3] and mesh [26,27,51,55,75,83,46], object pose [14,28,29,35,43,57,64,71,76,81], or both [6,17,18,19,69].”
Reconstructing Hand-Object Interactions in the Wild (2020, methodology): “Some approaches [58, 26, 12, 27] utilize neural network to predict the object shape, translation, and global orientation in the camera coordinate.”
Mesh R-CNN (2019, methodology): “Some methods predict the orientation [10, 20] or 3D pose [31, 44, 66] of known shapes.”
Deep Learning for LiDAR Point Clouds in Autonomous Driving: A Review (2020, background): “The orientation refers to the rigid transformation that aligns the detected object to its instance in the scene, which are the translations in each of the of x, y, and z directions as well as a rotation about each of these three axes [43, 44].”
Stereo R-CNN Based 3D Object Detection for Autonomous Driving (2019, methodology): “Recently an inverse-graphics framework [14] is proposed to predict both the 3D object pose and instance-level segmentation by graphic rendering and comparing.”
Soft Rasterizer: A Differentiable Renderer for Image-Based 3D Reasoning (2019, background): “able ray tracer to realize the differentiability of secondary rendering effects. Recent advances in 3D face reconstruction [38,40,39,41,9], material inference [27,7] and other 3D reconstruction tasks [49,37,33,14,22,34] have leveraged some other forms of differentiable rendering layers to obtain gra”
Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction (2019, methodology): “In particular, many learning-based techniques [57, 25, 37, 46, 45, 6, 21, 31, 10] have incorporated a differentiable rendering layer to enable an end-to-end architecture for 3D reconstruction and material inference in an unsupervised manner.”
Argoverse: 3D Tracking and Forecasting With Rich Maps (2019, methodology): “[30] Abhijit Kundu, Yin Li, and James M.”
ROI-10D: Monocular Lifting of 2D Detection to 6D Pose and Metric Shape (2018, methodology): “We initially used PCA to learn a low-dimensional shape, similar to [23].”
The ApolloScape Open Dataset for Autonomous Driving and Its Application (2018, background): “ing [13], [14], object 3D instance understanding [15], [16], [17], [18], [19] online in a self-driving video etc.”
Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild (2020, background): “The mesh models are obtained from [1,2,39] and are pre-processed to have fewer faces (about 1000) to make optimization more efficient.”
Geometry Uncertainty Projection Network for Monocular 3D Object Detection (2021, background): “The monocular 3D object detection aims to predict 3D bounding boxes from a single given image [13, 17, 21, 25, 31, 43].”
Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation (2022, methodology): “3DRCNN [28] and Mesh-RCNN [16] takes as input a single RGB image and predicts 3D mesh and pose of object instances in the image.”
BEHAVE: Dataset and Method for Tracking Human Object Interactions (2022, background): “Object reconstruction Most existing work on reconstructing 3D objects from RGB [21, 46, 53, 75, 78] and RGBD [45, 55, 82] data does so in isolation, without the human involvement or the interaction.”
Is Pseudo-Lidar needed for Monocular 3D Object detection? (2021, background): “These priors are often injected by aligning 2D keypoints with their projective 3D counterparts [6, 1, 2, 21, 52], or by learning a low-dimensional shape representation [28, 43].”
GDRNPP: A Geometry-Guided and Fully Learning-Based Object Pose Estimator (2021, positive): “Nevertheless, in contrast to [18], [84], we propose to let the network predict the allocentric representation [85] of rotation.”