Multiview Compressive Coding for 3D Reconstruction

By Chaozheng Wu, Justin Johnson, Jitendra Malik, Christoph Feichtenhofer, Georgia Gkioxari. 2023

97 citations recorded by the source corpus.

A central goal of visual recognition is to understand objects and scenes from a single image. 2D recognition has witnessed tremendous progress thanks to large-scale learning and general-purpose representations. Comparatively, 3D poses new challenges stemming from occlusions not depicted in the image. Prior works try to overcome these by inferring from multiple views or rely on scarce CAD models and category-specific priors which hinder scaling to novel settings. In this work, we explore single-view 3D reconstruction by learning generalizable representations inspired by advances in self-supervised learning. We introduce a simple framework that operates on 3D points of single objects or whole scenes coupled with category-agnostic large-scale training from diverse RGB-D videos. Our model, Multiview Compressive Coding (MCC), learns to compress the input appearance and geometry to predict the 3D structure by querying a 3D-aware decoder. MCC's generality and efficiency allow it to learn from large-scale and diverse data sources with strong generalization to novel objects imagined by DALLE2 or captured in-the-wild with an iPhone.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

Segment Anything (2023, methodology): “For example, MCC [106] can easily use SAM to segment an object of interest and achieve strong generalization to unseen objects for 3D reconstruction from a single RGB-D image.”
MVDream: Multi-view Diffusion for 3D Generation (2023, background): “It is pertinent to note ongoing research endeavors to reconstruct object shapes directly from monocular image inputs (Wu et al., 2023; Nichol et al., 2022; Jun & Nichol, 2023), aligning with the increasing stability of image generation techniques.”
LRM: Large Reconstruction Model for Single Image to 3D (2023, unknown): “Recent data-driven approach MCC (Wu et al., 2023) trains a generalizable transformer-based decoder with CO3D-v2 data (Reizenstein et al., 2021) to predict occupancy and color from the input image and its unprojected point cloud.”
Objaverse-XL: A Universe of 10M+ 3D Objects (2023, background): “Over the past year, generative AI has made its foray into 3D. MCC [46] learns a generalizable representation with self-supervised learning for 3D reconstruction.”
ImageDream: Image-Prompt Multi-view Diffusion for 3D Generation (2023, background): “Concurrently, there’s growing interest in reconstructing object shapes from monocular image inputs [14, 26, 48], demonstrating the evolving stability of image generation methodologies.”
Zero-1-to-3: Zero-shot One Image to 3D Object (2023, methodology): “Since MCC requires depth input, we use MiDaS [43, 42] off-the-shelf for depth estimation.”
Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers (2023, unknown): “MCC [68] also trains a transformer decoder to predict occupancy and color from RGB-D inputs.”
One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization (2023, background): “Recently, there has been an increasing number of work on learning to generate a 3D implicit ﬁeld from a single image [84, 45, 66, 24, 53, 17, 20, 26, 50, 78, 49].”
One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion (2023, background): “…directly from 3D synthetic data or real scans and generate various 3D representations such as point clouds [1, 15, 41, 47, 83], 3D voxels [9, 59, 74, 75], polygon meshes [16, 17, 26, 32, 37, 46, 68], parametric models [21], and implicit fields [8, 14, 19, 25, 30, 42, 48, 73, 78, 82, 84, 86, 87].”