RotationNet: Joint Object Categorization and Pose Estimation Using Multiviews from Unsupervised Viewpoints

By Asako Kanezaki, Yasuyuki Matsushita, Yoshifumi Nishida. 2016

526 citations recorded by the source corpus.

We propose a Convolutional Neural Network (CNN)-based model "RotationNet," which takes multi-view images of an object as input and jointly estimates its pose and object category. Unlike previous approaches that use known viewpoint labels for training, our method treats the viewpoint labels as latent variables, which are learned in an unsupervised manner during the training using an unaligned object dataset. RotationNet is designed to use only a partial set of multi-view images for inference, and this property makes it useful in practical scenarios where only partial views are available. Moreover, our pose alignment strategy enables one to obtain view-specific feature representations shared across classes, which is important to maintain high accuracy in both object categorization and pose estimation. Effectiveness of RotationNet is demonstrated by its superior performance to the state-of-the-art methods of 3D object classification on 10- and 40-class ModelNet datasets. We also show that RotationNet, even trained without known poses, achieves the state-of-the-art performance on an object pose estimation dataset.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Geology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Geology

Citations in the corpus

ScaleNet: An Unsupervised Representation Learning Method for Limited Information (2023, methodology): “Since RotNet only affects the input images of the ConvNet, it can be combined with other architectures such as GAN [7] or be used for 3D structured datasets [26].”
Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data (2019, methodology): “, using multiple view images [39, 48, 46, 22], or applying convolutions on 3D voxel grids [27, 43].”
Learning SO(3) Equivariant Representations with Spherical CNNs (2017, methodology): “Interestingly, RotationNet [30], which holds the current state-of-the-art on ModelNet40 classification, fails to generalize to unseen rotations, despite being multi-view based.”
View-GCN: View-Based Graph Convolutional Network for 3D Shape Analysis (2020, unknown): “View-GCN (AlexNet) also outperforms RotationNet (AlexNet) [28] by 2.6% in accuracy.”
Point2Sequence: Learning the Shape Representation of 3D Point Clouds with an Attention-based Sequence to Sequence Network (2018, methodology): “Inﬂuenced by the great success of 2D CNNs for images, such methods have achieved dominant results in 3D shape classiﬁcation and retrieval tasks (Kanezaki, Matsushita, and Nishida 2016).”
A Survey of Multi-View Representation Learning (2016, unknown): “10, multi-view images of a bus with 3D rotations (provided by [165]) are passed through a shared CNN (CNN1) separately, fused at a view-pooling layer, and then sent through the subsequent part”
View Inter-Prediction GAN: Unsupervised Representation Learning for 3D Shapes by Learning Global Shape Memories to Support Local View Predictions (2018, methodology): “For fair comparison, the result of VRN (Brock et al. 2016) is presented without ensemble learning, and the result of Ro-tationNet(Kanezaki, Matsushita, and Nishida 2018) is presented with views taken by the default camera system orientation that is identical to the others.”
PointCLIP: Point Cloud Understanding by CLIP (2021, methodology): “Therein, multiview methods project point clouds onto images of multiple views and process them with 2D Convolution Neural Networks (CNN) [22] pre-trained on ImageNet [28], such as MVCNN [49] and others [14, 15, 21, 26, 62].”
PRNet: Self-Supervised Learning for Partial-to-Partial Registration (2019, unknown): “DGCNN [22] 92.2 87.2 1024 3 VoxNet [64] 83.0 73.0 303 SubVolSup [65] 88.5 82.7 303 SubVolSup MO [65] 89.5 85.0 20 303 MVCNN 12x [66] 89.5 77.6 12 2242 MVCNN 80x [66] 90.2 86.0 80 2242 RotationNet 20x [67] 92.4 80.0 20 2242 Spherical CNNs [68] 88.9 86.9 2 642 PRNet (Ours) 85.2 80.5 1024 3 Table 4: Mo”
Deep Learning for LiDAR Point Clouds in Autonomous Driving: A Review (2020, methodology): “The most representative characteristic of RotationNet is that it treats viewpoints which are the observation of training images as latent variables.”
Review: deep learning on 3D point clouds (2020, methodology): “RotationNet [58] uses rotation to select the best viewpoint that maximizes the class likelihood; it leads the Modelnet40 [48] leaderboard at the time of this review.”
SPNet: Deep 3D Object Classification and Retrieval using Stereographic Projection (2018, methodology): “r arbitrary camera viewpoint. To perform pooling more eciently, Wang et al. [33] proposed a dominant set clustering technique where pooling is performed in each cluster individually. Kanezaki et al. [15] proposed RotationNet which takes multi-view images of an object and jointly estimates its object”
Learning Shape Templates With Structured Implicit Functions (2019, methodology): “Our network takes a stack of 20 depth images rendered from the vertices of a dodecahedron as input, as in [22].”
Point Transformer (2020, background): “Since shape information can be occluded by rendering point clouds from a specific viewpoint, multi-view approaches have been proposed that render multiple images from different angles [11], [12], [26], [27].”
PF-Net: Point Fractal Network for 3D Point Cloud Completion (2020, background): “However, multi-view and voxel-based representation[18, 7, 34, 11, 13, 37] leads to unnecessarily voluminous and limits the output resolution of voxels.”
Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline (2021, methodology): “Subsequent approaches include RotationNet which trains the network to also predict the viewpoint for each image (Kanezaki et al., 2018), GVCNN which groups features from subsets of views together before aggregating into a ﬁnal prediction (Feng et al., 2018), and hypergraph methods that consider the…”
Point Cloud GAN (2018, unknown): “1% RotationNet (Kanezaki et al., 2018) images 97 .”
3D2SeqViews: Aggregating Sequential Views for 3D Global Feature Learning by CNN With Hierarchical Attention Aggregation (2019, methodology): “In contrast, RotationNet [21] was proposed to learn global features by treating pose labels as latent variables which are optimized to self-align in an unsupervised manner.”
An End-to-End Transformer Model for 3D Object Detection (2021, methodology): “Projection methods [3, 19, 25, 26, 59, 60, 65] project 3D data into 2D planes and convert it into 2D grids.”