Viewpoints and keypoints

By Shubham Tulsiani, Jitendra Malik. 2014

404 citations recorded by the source corpus.

We characterize the problem of pose estimation for rigid objects in terms of determining viewpoint to explain coarse pose and keypoint prediction to capture the finer details. We address both these tasks in two different settings - the constrained setting with known bounding boxes and the more challenging detection setting where the aim is to simultaneously detect and correctly estimate pose of objects. We present Convolutional Neural Network based architectures for these and demonstrate that leveraging viewpoint estimates can substantially improve local appearance based keypoint predictions. In addition to achieving significant improvements over state-of-the-art in the above tasks, we analyze the error modes and effect of object characteristics on performance to guide future efforts towards this goal.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Control and Systems Engineering, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In 3D Vision & Pose

Citations in the corpus

StairNet: Top-Down Semantic Aggregation for Accurate One Shot Detection (2017, background): “Viewpoints and Keypoints [35] leverages the global viewpoint estimation to improve local keypoint predictions.”
Cross Modal Distillation for Supervision Transfer (2015, background): “…include almost all state-of-the-art systems: object detection [12], semantic segmentation [28], object segmentation [17], and pose estimation [38], which start from generic features that are learned on the ImageNet dataset [6] using over a million labeled images and specialize them for each…”
Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling (2018, methodology): “Researchers have also proposed numerous methods on 3D pose estimation [13, 42, 55, 58].”
End-to-End Recovery of Human Shape and Pose (2017, background): “classification problem by dividing the angles into bins [46].”
Multi-view Supervision for Single-View Reconstruction via Differentiable Ray Consistency (2017, background): “[32] extended similar ideas to more general categories and leveraged recognition systems [14, 17, 33] to automate test-time inference.”
Category-specific object reconstruction from a single image (2014, methodology): “For pose prediction, we use the CNNbased system of [34] and augment it to predict subtypes at test time.”
Single Image 3D Interpreter Network (2016, methodology): “VDPM [17] and DPM-VOC+VP [50] are trained on PASCAL VOC 2012, V&K [18] is trained on PASCAL 3D+, Su et al .”
Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Views (2015, methodology): “For more details of the metric definition, please refer to [35].”
PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes (2017, methodology): “ features are extracted from either points of interest or every pixel in the image and matched to features on the 3D models to establish the 2D3D correspondences, from which 6D poses can be recovered [20, 25, 30, 22]. Feature-based methods are able to handle occlusions between objects. However, they”
Learning 3D Object Categories by Looking Around Them (2017, methodology): “Throughout the experimental section, we consider three datasets for training and benchmarking our network: (1) FreiburgCars (FrC) [35] which consists of RGB video sequences with the camera circling around various types of cars; (2) the Large Dataset of Object Scans (LDOS) [6] containing RGBD sequenc”
Real-time Deep Pose Estimation with Geodesic Loss for Image-to-Template Rigid Registration (2018, background): “the orientation [17], [18]; and 2) models that predict the pose of the object directly from images [19], [20].”
HybridPose: 6D Object Pose Estimation Under Hybrid Representations (2020, background): “While early works typically formulate pose estimation as end-to-end pose classification [39] or pose regression [16, 42], recent pose estimation methods usually leverage keypoints as an intermediate representation [38, 34], and align predicted 2D keypoints with ground-truth 3D keypoints.”
DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion (2019, methodology): “Other methods propose to directly estimate objects pose from images using CNN-based architectures [28, 36].”
PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimation (2017, methodology): “[33] uses R-CNN to propose 2D RoI and another network to regress the object poses.”
6-DoF object pose from semantic keypoints (2017, methodology): “, [22], [23], [24], [25]) has attracted considerable study, while limited attention has been given to their application with generic object categories [26], [20].”
DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision (2021, background): “to reduce uncertainties through additional constraints, such as determining camera poses and viewpoints [31, 36].”
StarMap for Category-Agnostic Keypoint and Viewpoint Estimation (2018, methodology): “Tulsiani et al. [37] introduced viewpoint estimation as a bin classiﬁcation prob-4 lem for each viewing angle (azimuth, elevation and in-plane rotation).”
PVNet: Pixel-Wise Voting Network for 6DoF Pose Estimation (2018, methodology): “To avoid this problem, [38, 33, 23, 35] discretize the rotation space and cast the 3D rotation estimation into a classification task.”
EuroCity Persons: A Novel Benchmark for Person Detection in Traffic Scenes (2018, background): “In [64] and [65], orientation estimation is handled as a multi-class classification problem.”
Recurrent Network Models for Human Dynamics (2015, methodology): “We represent K body joint locations as a set of K N×N heat maps over the person’s bounding box, that represent likelihood for each joint to appear in each of the N(2) grid locations, similar to [38].”