Pairwise Decomposition of Image Sequences for Active Multi-view Recognition

By Edward Johns, Stefan Leutenegger, Andrew J. Davison. 2016

237 citations recorded by the source corpus.

A multi-view image sequence provides a much richer capacity for object recognition than from a single image. However, most existing solutions to multi-view recognition typically adopt hand-crafted, model-based geometric methods, which do not readily embrace recent trends in deep learning. We propose to bring Convolutional Neural Networks to generic multi-view recognition, by decomposing an image sequence into a set of image pairs, classifying each pair independently, and then learning an object classifier by weighting the contribution of each pair. This allows for recognition over arbitrary camera trajectories, without requiring explicit training over the potentially infinite number of camera paths and lengths. Building these pairwise relationships then naturally extends to the next-best-view problem in an active recognition framework. To achieve this, we train a second Convolutional Neural Network to map directly from an observed image to next viewpoint. Finally, we incorporate this into a trajectory optimisation task, whereby the best recognition confidence is sought for a given trajectory length. We present state-of-the-art results in both guided and unguided multi-view recognition on the ModelNet dataset, and show how our method can be used with depth images, greyscale images, or both.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Generative and Discriminative Voxel Modeling with Convolutional Neural Networks (2016, methodology): “71% accuracy, which is also better than any previously published result; we report our ModelNet10 accuracy for models trained on ModelNet40 for consistency with previous results[2][3][4].”
3D Simulation for Robot Arm Control with Deep Q-Learning (2016, neutral): “Simulation for robotics has seen success in tasks such as guided control of a camera for object recognition using depth image simulation [16], and grasp pose estimation using physics simulation [17].”
FusionNet: 3D Object Classification Using Multiple Data Representations (2016, methodology): “Recent work on Active Multi-View Recognition [10] predicts the Next Best View (NBV) which is most likely to give the highest extra information about the object, needing a smaller number of image sequences during test-time to predict the model class.”
Deep learning a grasp function for grasping under gripper pose uncertainty (2016, background): “alisation is moving away from using hand-engineered features [29] and towards deep learning features [31], active recognition has achieved state-of-the-art performance by deep learning camera control [2], deep reinforcement learning is enabling end-to-end training for robot arm control [25], and eve”
Fine-Pruning: Joint Fine-Tuning and Compression of a Convolutional Network with Bayesian Optimization (2017, methodology): “Convolutional neural networks (CNNs) have been widely adopted for visual analysis tasks such as image classiﬁcation [14, 26], object detection [17, 22], action recognition [25, 29], place recognition [1, 35], 3D shape classiﬁcation [12, 19], image colorization [34], and camera pose estimation [13].”
3D Object Classification via Spherical Projections (2017, background): “[10] provides a different way to fine-tune the network with rendered images, each of which is given a weight to measure its importance to the final prediction.”
Multi-view Harmonized Bilinear Network for 3D Object Recognition (2018, methodology): “Analogous to [32, 15], we evaluate the influence of additional depth modality as well.”
Emergence of exploratory look-around behaviors through active observation completion (2019, methodology): “Because camera motions are restricted to have proximity to the current camera angle and candidate viewpoints partially overlap, the discrete action space promotes efficiency without neglecting the physical realities of the problem [following (8, 9, 23, 26)].”
Learning to Look Around: Intelligently Exploring Unseen Environments for Unknown Tasks (2017, methodology): “We discretize the space of all viewpoints into a “viewgrid” V (X), as in [29,31,43].”
SAR Automatic Target Recognition Based on Multiview Deep Learning Framework (2018, background): “In recent years, researchers have studied the recognition problems with multiview images, and have concluded that a multi-view image sequence could provide much richer classiﬁcation information than a single image [26].”
Deep Learning Advances in Computer Vision with 3D Data (2017, methodology): “%) and ModelNet10 (i.e., 97.14%).”
Sidekick Policy Learning for Active Visual Exploration (2018, methodology): “Similarly, an active object recognition system [29,31,2,65,28] can only see its previously selected views of the object; yet if trained with CAD models, it could observe all possible views while learning.”
Triplet-Center Loss for Multi-view 3D Object Retrieval (2018, background): “, [33, 14, 25], only a few novel deep learning based approaches specifically designed for 3D object”
Structured Prediction of Unobserved Voxels from a Single Depth Image (2016, background): “As new data arrives, a next-best-view algorithm [42, 31] could leverage our predictions to guide the camera to a position which captures the geometry most likely to be informative for verifying our proposals.”
Robotic grasp detection using deep convolutional neural networks (2016, background): “For example, robot localization is moving from using hand-engineered features [13] to deep learning features [14], deep reinforcement learning is being used for end-to-end training for robotic arm control [15], multi-view object recognition has achieved state-of-the-art performance by deep learning ”
General-Purpose Deep Point Cloud Feature Extractor (2018, background): “Another avenue researchers frequently choose is to map the 3D data to 2D, either by creating images from multiple angles [48, 5, 24, 60], or by creating a mapping function from mesh to image such that depth information is preserved [44, 47].”
SPNet: Deep 3D Object Classification and Retrieval using Stereographic Projection (2018, background): “[14] classified each pair independently and learned an object classifier by weighting the contribution of each pair, which allows 3D shape recognition over arbitrary camera viewpoint.”
ShapeCodes: Self-supervised Feature Learning by Lifting Views to Viewgrids (2017, methodology): “ModelNet is the most widely used dataset in recent 3D object categorization work [31,33,37,38,61,62].”