Learning a Predictable and Generative Vector Representation for Objects

By Rohit Girdhar, David F. Fouhey, Mikel D. Rodriguez, Abhinav Kumar Gupta. 2016

747 citations recorded by the source corpus.

What is a good vector representation of an object? We believe that it should be generative in 3D, in the sense that it can produce new 3D objects; as well as be predictable from 2D, in the sense that it can be perceived from 2D images. We propose a novel architecture, called the TL-embedding network, to learn an embedding space with these properties. The network consists of two components: (a) an autoencoder that ensures the representation is generative; and (b) a convolutional network that ensures the representation is predictable. This enables tackling a number of tasks including voxel prediction from 2D images and 3D model retrieval. Extensive experimental analysis demonstrates the usefulness and versatility of this embedding.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Descriptor Networks for 3D Shape Synthesis and Analysis (2018, background): “Recently, with the introduction of large 3D CAD datasets, e.g., ShapeNet [29, 4], some interesting attempts [5, 24, 17] have been made on object recognition and synthesis based on voxelized 3D shape data.”
Multiview Compressive Coding for 3D Reconstruction (2023, background): “One line of work trains models that predict 3D geometry via CAD [17, 68], meshes [31,75], voxels [16,72] or point clouds [13,37] supervision.”
Occupancy Networks: Learning 3D Reconstruction in Function Space (2018, background): “Voxel Representations: Due to their simplicity, voxels are the most commonly used representation for discriminative [45,55,63] and generative [9,23,58,64,75,77] 3D tasks.”
OctNet: Learning Deep 3D Representations at High Resolutions (2016, background): “Interestingly, these lowdimensional representations can be directly inferred from a single image [14] or a sequence of images [8].”
Deep Level Sets: Implicit Surface Representations for 3D Shape Inference (2019, methodology): “Recently, [12] and [9] proposed the use of shape and camera features along with the images, respectively.”
Learning to Reconstruct Shapes from Unseen Classes (2018, background): “…et al., 2015] and deep networks have contributed to the significant progress in recent years, mostly with voxel representations [Choy et al., 2016, Girdhar et al., 2016, Häne et al., 2017, Kar et al., 2015, Novotny et al., 2017, Rezende et al., 2016, Tatarchenko et al., 2016, Tulsiani et al.,…”
Learning Implicit Fields for Generative Shape Modeling (2018, background): “There have been a variety of 3D shape representations for deep learning of shapes, such as voxel grids [9, 14, 28, 45, 46], octrees [18, 35, 40, 43, 44], multi-view images [29, 39], point clouds [1, 12, 13, 32, 33, 47, 48], geometry images [37, 38], deformable mesh/patches [16, 38, 42, 47], and part”
Learning Shape Priors for Single-View 3D Completion and Reconstruction (2018, methodology): “With the development of large-scale shape repositories like ShapeNet [5] and methods like deep convolutional networks, researchers have built more scalable and eﬃcient models in recent years [7,13,18,27,36,38,48,52,56,57,62].”
RayNet: Learning Volumetric 3D Reconstruction with Ray Potentials (2018, background): “Learning-based 3D Reconstruction: The development of large 3D shape databases [4, 38] has fostered the development of learning based solutions [3, 5, 8, 37] to 3D reconstruction.”
Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling (2018, background): “Using deep convolutional networks, researchers have made significant progress in recent years [9, 17, 21, 29, 41, 43, 56, 59, 60, 62, 66, 52, 61].”
Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images (2018, background): “Restricted by the prevalent grid-based deep learning architectures, most works [6, 11] outputs 3D voxels, which are usually with low resolutions due to the memory”
Learning Category-Specific Mesh Reconstruction from Image Collections (2018, methodology): “Relying on ground-truth 3D supervision (using synthetic data), some approaches have examined learning voxel [4,8,39,33], point cloud [7] or octree [10,26] prediction.”
A Papier-Mache Approach to Learning 3D Surface Generation (2018, neutral): “Methods for generating a voxel grid have been demonstrated with various inputs, namely one or several images [6, 9], full 3D objects in the form of voxel grids [9, 33], and 3D objects with missing shape parts [33, 11].”
MarrNet: 3D Shape Reconstruction via 2.5D Sketches (2017, methodology): “The detailed encoder structure can be found in Girdhar et al. [2016].”
Learning a Multi-View Stereo Machine (2017, background): “Approaches which predict shape from a single image have been proposed in form of direct depth map regression [46, 31, 10], generating multiple depth maps from novel viewpoints [51], producing voxel occupancies [5, 16], geometry images [49] and point clouds [11].”
Multi-view Supervision for Single-View Reconstruction via Differentiable Ray Consistency (2017, methodology): “[5, 13], we train a CNN on rendered ShapeNet images using cross entropy loss with the ground-truth CAD model.”
OctNetFusion: Learning Depth Fusion from Data (2017, background): “Generalization on Unseen Categories: Most existing approaches that leverage deep learning for 3D reconstruction train a model specific for a particular object class [4, 8, 17, 41, 35] which typically does not generalize well to other views=10 views=20”
Hierarchical Surface Prediction for 3D Object Reconstruction (2017, background): “Recently, 3D prediction approaches which directly learn a function to map an input image to the output geometry have emerged [6], [13], [35].”
Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs (2017, methodology): “Applications include single- and multi-view 3D object reconstruction trained in supervised [14, 5, 7, 17] and unsupervised [37, 12, 19] ways, probabilistic generative modeling of 3D shapes [36, 35, 29], semantic segmentation [2, 4] and shape deformation [39].”
Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling (2016, methodology): “Following previous work [5], we test it on all six categories of the IKEA dataset [11]: bed, bookcase, chair, desk, sofa, and table.”