FusionNet: 3D Object Classification Using Multiple Data Representations

By Vishakh Hegde, Reza Bosagh Zadeh. 2016

220 citations recorded by the source corpus.

High-quality 3D object recognition is an important component of many vision and robotics systems. We tackle the object recognition problem using two data representations, to achieve leading results on the Princeton ModelNet challenge. The two representations: 1. Volumetric representation: the 3D object is discretized spatially as binary voxels - $1$ if the voxel is occupied and $0$ otherwise. 2. Pixel representation: the 3D object is represented as a set of projected 2D pixel images. Current leading submissions to the ModelNet Challenge use Convolutional Neural Networks (CNNs) on pixel representations. However, we diverge from this trend and additionally, use Volumetric CNNs to bridge the gap between the efficiency of the above two representations. We combine both representations and exploit them to learn new features, which yield a significantly better classifier than using either of the representations in isolation. To do this, we introduce new Volumetric CNN (V-CNN) architectures.

Subfields: Computer Vision and Pattern Recognition, Geology, Multimodal Learning, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Geology

In Multimodal Learning

In 3D Vision & Pose

Citations in the corpus

OctNet: Learning Deep 3D Representations at High Resolutions (2016, background): “While performance gains can be obtained using orthogonal approaches such as network ensembles [5] or a combination of 3D and 2D convolutional networks [19,41], in this paper we deliberately focus on “pure” 3D convolutional network approaches to isolate the effect of resolution from other influencing”
3D fully convolutional network for vehicle detection in point cloud (2016, methodology): “The approach proposed in this paper is inspired by the basic idea of DenseBox.”
Multi-view 3D Object Detection Network for Autonomous Driving (2016, background): “For outdoor scenes, Vote3D [22] places 3D windows with different sizes and orientations in 3D space.”
Generative and Discriminative Voxel Modeling with Convolutional Neural Networks (2016, methodology): “Compared to FusionNets[4], our model requires significantly fewer parameters (18M as opposed to 118M for a full FusionNet) and fewer object views (12 or 24, compared to 60).”
Orientation-boosted Voxel Nets for 3D Object Recognition (2016, background): “Quite recently and parallel to this work, there have been several works utilizing 2D or 3D CNNs merely on 3D CAD models of ModelNet [38] in supervised [2, 9, 13], semisupervised [24] and unsupervised [37] fashions.”
CubeNet: Equivariance to 3D Rotation and Translation (2018, neutral): “…( × 10 6 , 2 s.f.) 3D ShapeNets [41] 0.8354 12 Xu & Todovoric [48] 0.8800 0.080 3D-GAN [49] 0.9100 11 VRN [42] 0.9133 18 VoxNet [40] 0.9200 0.92 Fusion-Net [50] 0.9311 120 ORION [43] 0 For the single-model category, our four-group, rotation-averaged network attains state-of-the-art performance.”
MeshNet: Mesh Neural Network for 3D Shape Representation (2018, methodology): “FusionNet (Hegde and Zadeh 2016) uses the volumetric grid and multiview for classification.”
3DmFV: Three-Dimensional Point Cloud Classification in Real-Time Using Convolutional Neural Networks (2018, unknown): “…3DShapeNets [4] 83.5 77.32 VoxNet [3] 92.0 83.0 OctNet [21] 90.9 86.5 O-CNN [22] - 89.9 VRN [20] 93.6 91.33 VRN ensemble [20] 97.14 95.54 FusionNet [44] 93.1 90.8 PointNet [30] - 89.2 a PointNet++ [31] - 90.7 a Kd-network [32] 94.0 b / 93. of 14 object classes with a total of 588 objects over four…”
SPLATNet: Sparse Lattice Networks for Point Cloud Processing (2018, background): “FusionNet [18] combines shape classification scores from a volumetric and a multi-view network, yet this fusion happens at a late stage, after the final fully connected layer of these networks, and does not jointly consider their intermediate local and global feature representations.”
Escape from Cells: Deep Kd-Networks for the Recognition of 3D Point Cloud Models (2017, background): “Importantly, the structure of the kd-tree serves as a shape descriptor (e.g. ‘ones’ are dominated by vertical splits, and ‘zeroes’ tend to interleave vertical and horizontal splits as a kd-tree is traversed from the root to a leaf).”
PVNet: A Joint Convolutional Network of Point Cloud and Multi-View for 3D Shape Recognition (2018, background): “0% (13)FusionNet [12] ImageNet1K ModelNet40 Volumetric and Multi-View 90.”
3D Object Classification via Spherical Projections (2017, methodology): “MVCNN-MultiRes [17], 3D ShapeNets [27], Voxnet [13], FusionNet [6], Volumetric CNN [17] and OctNet [20].”