Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction

By Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, L. Sbordone, Patrick Labatut, David Novotný. 2021

744 citations recorded by the source corpus.

Traditional approaches for learning 3D object categories have been predominantly trained and evaluated on synthetic datasets due to the unavailability of real 3D-annotated category-centric data. Our main goal is to facilitate advances in this field by collecting real-world data in a magnitude similar to the existing synthetic counterparts. The principal contribution of this work is thus a large-scale dataset, called Common Objects in 3D, with real multi-view images of object categories annotated with camera poses and ground truth 3D point clouds. The dataset contains a total of 1.5 million frames from nearly 19,000 videos capturing objects from 50 MS-COCO categories and, as such, it is significantly larger than alternatives both in terms of the number of categories and objects.We exploit this new dataset to conduct one of the first large-scale "in-the-wild" evaluations of several new-view-synthesis and category-centric 3D reconstruction methods. Finally, we contribute NerFormer - a novel neural rendering method that leverages the powerful Transformer to reconstruct an object given a small number of its views.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Evaluation & Benchmarks, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

In 3D Vision & Pose

Citations in the corpus

Multiview Compressive Coding for 3D Reconstruction (2023, methodology): “For a fair comparison with MCC, we extend baselines [21,51] with depth supervision [9] or using depth as input.”
Urban Radiance Fields (2021, result): “This results in parts of the scene being observed by only a small number of cameras, in contrast to other datasets [34,43,50,57] which capture scenes uniformly with a large number of cameras.”
CAT3D: Create Anything in 3D with Multi-View Diffusion Models (2024, methodology): “We trained the multi-view diffusion model at the core of CAT3D on four datasets with camera pose annotations: Objaverse [75], CO3D [74], RealEstate10k [76] and MVImgNet [77].”
Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model (2023, methodology): “Most of previous works (Reizenstein et al., 2021; Trevithick & Yang, 2021; Yu et al., 2021) are either trained on a small-scale dataset such as ShapeNet, or trained in a category-specific manner.”
CLIP-Mesh: Generating textured meshes from text using pretrained image-text models (2022, methodology): “For example, datasets such as Shapenet [Chang et al. 2015] and CO3D [Reizenstein et al. 2021] provide 50 object categories respectively.”
ReconFusion: 3D Reconstruction with Diffusion Priors (2023, neutral): “GeNVS [5] and SparseFusion [16] train on real-world multiview data [39] and further incorporate 3D geometry priors by conditioning on rendered features [52, 67].”
3rd Continual Learning Workshop Challenge on Egocentric Category and Instance Level Object Understanding (2022, background): “The most relevant to our dataset are those containing daily life objects, such as Objectron [55], CO3D [56], Epic-Kitchens [57], and Ego4D [58].”
Continuous 3D Perception Model with Persistent State (2025, neutral): “Examples of our datasets include CO3Dv2 [60], ARK-itScenes [5], ScanNet++ [95], TartanAir [87], Waymo [75], MegaDepth [41], MapFree [3], DL3DV [44], and Dynam-icStereo [35].”
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives (2023, neutral): “Multiview and ego-exo datasets Most existing multi-view datasets focus on static scenes [20, 128, 151, 175, 176] and objects [133, 173], with limited (exo only) multi-view human activity [26, 169].”
LRM: Large Reconstruction Model for Single Image to 3D (2023, positive): “…on approximately one million 3D shapes and video data across diverse categories (Deitke et al., 2023; Yu et al., 2023); this is substantially larger than recent methods that apply relatively shallower networks and smaller datasets (Chang et al., 2015; Reizenstein et al., 2021; Downs et al., 2022).”
Generative Novel View Synthesis with 3D-Aware Diffusion Models (2023, background): “A8 provides additional visual comparisons on the ShapeNet [2] dataset against baselines.”
ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis (2024, methodology): “We extensively evaluate our method for zero-shot novel view synthesis and sparse view 3D-GS reconstruction on various datasets, including Tanks-and-Temples [21], RealEstate10K [7], and CO3D [22].”
DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Vision (2023, neutral): “Object-level datasets like Objectron [1], CO3D [21], and MVimgnet [39] offer substantial scale for learning-based reconstruction.”
OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generation (2023, methodology): “A branch of works [11, 51, 73, 90, 104] has also explored the generalization ability of NeRF-based frameworks, where they aim to learn priors from deep image features across multiple scenes.”
RealFusion 360° Reconstruction of Any Object from a Single Image (2023, methodology): “Examples include IBRNet [51], NeRF-WCE [13], PixelNeRF [60], NeRFormer [38], and ViewFormer [22].”
Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Pass (2025, positive): “On CO3Dv2 [39], Fast3R gets 99.7% accuracy within 15-degrees for pose estimation, over a 14x error reduction compared to DUSt3R with global alignment.”
ABO: Dataset and Benchmarks for Real-World 3D Object Understanding (2021, background): “CO3D [46] also offers videos of common objects from 50 different categories, however they do not provide full 3D mesh reconstructions.”
Zero-1-to-3: Zero-shot One Image to 3D Object (2023, background): “Except for this watermark, it is identical to the accepted version; the final published version of the proceedings is available on IEEE Xplore. in the direction of open-world 3D reconstruction by pre-training on large-scale, diverse datasets such as CO3D [44, 31, 36, 16].”
ProcTHOR: Large-Scale Embodied AI Using Procedural Generation (2022, background): “Additionally, we plan to scale up our asset databases by leveraging many open-source 3D asset databases, such as ABO [26], PartNet [81], ShapeNet [15], Google Scanned Objects [30], and CO3D [97], among others.”