Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling

By Xingyuan Sun, Jiajun Wu, Xiuming Zhang, Zhoutong Zhang, Chengkai Zhang, Tianfan Xue, Josh Tenenbaum, William T. Freeman. 2018

524 citations recorded by the source corpus.

We study 3D shape modeling from a single image and make contributions to it in three aspects. First, we present Pix3D, a large-scale benchmark of diverse image-shape pairs with pixel-level 2D-3D alignment. Pix3D has wide applications in shape-related tasks including reconstruction, retrieval, viewpoint estimation, etc. Building such a large-scale dataset, however, is highly challenging; existing datasets either contain only synthetic data, or lack precise alignment between 2D images and 3D shapes, or only have a small number of images. Second, we calibrate the evaluation criteria for 3D shape reconstruction through behavioral studies, and use them to objectively and systematically benchmark cutting-edge reconstruction algorithms on Pix3D. Third, we design a novel model that simultaneously performs 3D reconstruction and pose estimation; our multi-task learning approach achieves state-of-the-art performance on both tasks.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Geology, Evaluation & Benchmarks, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Geology

In Evaluation & Benchmarks

In 3D Vision & Pose

Citations in the corpus

Multiview Compressive Coding for 3D Reconstruction (2023, background): “Others [17, 68] predict 3D from a single image but rely on expensive CAD supervision [6, 60].”
Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, background): “One highly related direction is to model and estimate 3D hand joints [5,44,45,53,60,63,66,72,78,79,80,85,3] and mesh [26,27,51,55,75,83,46], object pose [14,28,29,35,43,57,64,71,76,81], or both [6,17,18,19,69].”
Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding (2020, methodology): “…evaluate sim-to-real transfer performance on two scene understanding tasks – semantic segmentation on NYUv2 [68] and 3D shape prediction on Pix3D [73] – where we ﬁnd that pre-training on our dataset signiﬁcantly improves performance on both tasks, and achieves state-of-the-art performance on the…”
Occupancy Networks: Learning 3D Reconstruction in Function Space (2018, background): “An additional quantitative evaluation on the Pix3D dataset [65] can be found in the supplementary.”
Learning to Reconstruct Shapes from Unseen Classes (2018, methodology): “Besides ShapeNet renderings, we also test these models, trained only on synthetic data, on real images from Pix3D [Sun et al., 2018], a dataset of real images and the ground-truth shape of every pictured object.”
Learning Shape Priors for Single-View 3D Completion and Reconstruction (2018, methodology): “Table 1 also suggests the naturalness loss improve the completion results, achieving comparable IoU scores and better (lower) CDs. CD has been reported to be better at capturing human perception of shape quality [45].”
SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation (2022, methodology): “Next, we assess 3D shape reconstruction from a single image on the real-world benchmark Pix3D [43] dataset.”
Reconstructing Hand-Object Interactions in the Wild (2020, methodology): “On the other hand, some approaches [24, 28, 53, 57, 45, 39] assume known 3D object model and focus on 6DOF object pose prediction.”
ABO: Dataset and Benchmarks for Real-World 3D Object Understanding (2021, methodology): “The IKEA [34] and Pix3D [53] datasets sought to improve upon this by annotating real images with exact, pixelaligned 3D models.”
Maintaining Natural Image Statistics with the Contextual Loss (2018, methodology): “zing Eq. (8) is equivalent to minimizing the asymmetric Chamfer Distance [22] between X;Y dened as CD(X;Y) = 1 jXj X i min j dist(x i;y j) (10) CD has been previously used mainly for shape retrieval [23,24] were the points are in R3. For each point in set X, CD nds the nearest point in the set Y and”
Visual Object Networks: Image Generation with Disentangled 3D Representations (2018, methodology): “Recent and concurrent work has learned to infer both texture and 3D shapes from 2D images, represented as parametrized meshes [Kanazawa et al., 2018], point clouds [Tatarchenko et al., 2016], or colored voxels [Tulsiani et al., 2017, Sun et al., 2018b].”
3D Shape Perception from Monocular Vision, Touch, and Shape Priors (2018, unknown): “In particular, the vision component of our model builds upon MarrNet [19], which jointly estimates intrinsic images and full 3D shape from a color image and has demonstrated good performance on standard benchmarks [20].”
Pix2Vox: Context-Aware 3D Reconstruction From Single and Multi-View Images (2019, methodology): “267 Pix3D (w/ Pose) [22] 0.”
BEHAVE: Dataset and Method for Tracking Human Object Interactions (2022, methodology): “Given template object meshes and annotated 2D keypoints from multi-views, we register the mesh to the images by optimizing the reprojection loss, similar to the method used in Pix3D registration [7].”
Few-Shot Object Detection and Viewpoint Estimation for Objects in the Wild (2020, methodology): “Apart from these two datasets, we also evaluate our method on a more recent benchmark: Pix3D [92].”
Mesh R-CNN (2019, methodology): “We then evaluate our full Mesh R-CNN for the task of 3D shape prediction in the wild on the challenging Pix3D dataset [60].”
Mask2CAD: 3D Shape Prediction by Learning to Segment and Retrieve (2020, methodology): “Experiments on unseen shapes of the Pix3D dataset [38] show notable improvement when we have access to all CAD models at test time (but no access to corresponding RGB images of the”
AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generation (2022, unknown): “benchmark Pix3D [27], using cropped and segmented images as input for reconstruction.”