Pix2Vox: Context-Aware 3D Reconstruction From Single and Multi-View Images

By Haozhe Xie, Hongxun Yao, Xiaoshuai Sun, Shangchen Zhou, Shengping Zhang, Xiaojun Tong. 2019

386 citations recorded by the source corpus.

Recovering the 3D representation of an object from single-view or multi-view RGB images by deep neural networks has attracted increasing attention in the past few years. Several mainstream works (e.g., 3D-R2N2) use recurrent neural networks (RNNs) to fuse multiple feature maps extracted from input images sequentially. However, when given the same set of input images with different orders, RNN-based approaches are unable to produce consistent reconstruction results. Moreover, due to long-term memory loss, RNNs cannot fully exploit input images to refine reconstruction results. To solve these problems, we propose a novel framework for single-view and multi-view 3D reconstruction, named Pix2Vox. By using a well-designed encoder-decoder, it generates a coarse 3D volume from each input image. Then, a context-aware fusion module is introduced to adaptively select high-quality reconstructions for each part (e.g., table legs) from different coarse 3D volumes to obtain a fused 3D volume. Finally, a refiner further refines the fused 3D volume to generate the final output. Experimental results on the ShapeNet and Pix3D benchmarks indicate that the proposed Pix2Vox outperforms state-of-the-arts by a large margin. Furthermore, the proposed method is 24 times faster than 3D-R2N2 in terms of backward inference time. The experiments on ShapeNet unseen 3D categories have shown the superior generalization abilities of our method.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

Multiview Compressive Coding for 3D Reconstruction (2023, methodology): “Supervised approaches predict voxels [67, 74] or meshes [69, 71] by training deep nets.”
GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis (2020, background): “Key advantages over voxel [6,9,50,52, 65,66] or mesh-based [15,26,43,62] methods are that they do not discretize space and are not restriced in topology.”
PGSR: Planar-Based Gaussian Splatting for Efficient and High-Fidelity Surface Reconstruction (2024, neutral): “Numerous pioneering efforts have leveraged pure deep neural networks to predict surface models directly from single or multiple image conditions using point clouds [43], [44], voxels [45], [46], and triangular meshes [7], [47] or implicit ﬁelds [48], [49] in end-to-end manner.”
AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generation (2022, neutral): “Baselines We compare with the following methods: • Pix2Vox [43]: a state-of-the-art approach for 3D reconstruction.”
Artificial intelligence in the creative industries: a review (2020, methodology): “The state-of-the-art methods for volumetric and surface-based representations are Pix2Vox [205] and AllVPNet [211] reporting an Intersection-over-Union (IoU) measure of 0.”
NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction (2021, background): “These inductive biases can be explicit representations, such as voxel grids [13, 5, 44], point cloud [7, 24, 18], meshes [41, 43, 14], and implicit representations.”
SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation (2022, neutral): “Input Output Input Output Ours AutoSDF [25] ResNet2Vox ResNet2SDF Pix2Vox [46]”
Neural Radiance Flow for 4D View Synthesis and Video Processing (2020, positive): “Recently, neural continuous implicit fields [9, 17, 35, 41, 49, 51, 58, 61, 75] have been developed to address the discretization issues and limited resolution of classical 3D representations such as voxel grids [5, 11, 40, 56, 70, 71, 74], point clouds [13, 14, 54, 55] and meshes [18, 26, 28, 34, 6”
Multi-view 3D Reconstruction with Transformers (2021, methodology): “To obtain the reported best results, Pix2Vox-A and Pix2Vox++/A both adopt an additional 3DCNN-based refiner containing another 3D-CNN and 3DDCNN.”
GRNet: Gridding Residual Network for Dense Point Cloud Completion (2020, background): “It follows the idea of a 3D encoder-decoder with U-net connections [44,45].”
ABO: Dataset and Benchmarks for Real-World 3D Object Understanding (2021, methodology): “Methods that require full shape supervision in the single-view [13, 17, 41, 53, 66] and multi-view [7, 23, 61] case are often trained using ShapeNet.”
DUSt3R: Geometric 3D Vision Made Easy (2023, neutral): “Shapes can be stored as occupancy in regular grids [19, 98, 112, 118, 138, 154, 155], octree structures [128], collections of parametric surface elements [39], point clouds encoders [29, 66, 67], free-form deformation of template meshes [89] or per-view depthmaps [54].”
One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion (2023, background): “…directly from 3D synthetic data or real scans and generate various 3D representations such as point clouds [1, 15, 41, 47, 83], 3D voxels [9, 59, 74, 75], polygon meshes [16, 17, 26, 32, 37, 46, 68], parametric models [21], and implicit fields [8, 14, 19, 25, 30, 42, 48, 73, 78, 82, 84, 86, 87].”
Atlas: End-to-End 3D Scene Reconstruction from Posed Images (2020, background): “Pix2Vox [56] encodes each image to a latent code and then decodes a voxel representation for each and then fuses them.”
One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization (2023, methodology): “By combing 2D image encoder and 3D generators, they generates 3D data in various representations, including 3D voxels [18, 79, 10, 81, 80, 85], point clouds [15, 88, 19, 1, 44, 90], polygon meshes [30, 73, 77, 51], and parametric models [54, 94, 95].”
Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era (2019, background): “To overcome these limitations, Xie et al. [86] introduced Pix2Vox, which is composed of multiple encoder-decoder blocks, running in parallel, each one predicts a coarse volumetric grid from its input frame.”