Generic 3D Representation via Pose Estimation and Matching

By Amir Zamir, Tilman Wekel, Pulkit Agrawal, Colin Wei, Jitendra Malik, Silvio Savarese. 2016

98 citations recorded by the source corpus.

Though a large body of computer vision research has investigated developing generic semantic representations, efforts towards developing a similar representation for 3D has been limited. In this paper, we learn a generic 3D representation through solving a set of foundational proxy 3D tasks: object-centric camera pose estimation and wide baseline feature matching. Our method is based upon the premise that by providing supervision over a set of carefully selected foundational tasks, generalization to novel tasks and abstraction capabilities can be achieved. We empirically show that the internal representation of a multi-task ConvNet trained to solve the above core problems generalizes to novel 3D tasks (e.g., scene layout estimation, object pose estimation, surface normal estimation) without the need for fine-tuning and shows traits of abstraction abilities (e.g., cross modality pose estimation).

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Data-Efficient Image Recognition with Contrastive Predictive Coding (2019, background): “, 2015; Jayaraman & Grauman, 2015), scene geometry (Zamir et al., 2016), or sound (Arandjelovic & Zisserman, 2017; 2018) can all serve as natural sources of supervision.”
Multi-task Self-Supervised Visual Learning (2017, methodology): “Tasks that use auxiliary information such as multi-modal information beyond pixels include: predicting sound given videos [26], predicting camera motion given two images of the same scene [1, 17, 44], or predicting what robotic motion caused a change in a scene [2, 29, 30, 31, 32].”
Time-Contrastive Networks: Self-Supervised Learning from Video (2017, methodology): “generation of each modality (i.e. pixel-level reconstruction). [5,33] use the spatial coherence in images as a self-supervision signal, [17] use motion cues to self-supervise a segmentation task, and [34] use camera pose estimation and feature matching. These methods are more focused on spatial rela”
OctNet: Learning Deep 3D Representations at High Resolutions (2016, background): “While 2D convolutional networks have proven very successful in extracting information from images [11, 13, 18, 35, 41, 42, 46, 48, 49], there exists comparably little work on processing three-dimensional data.”
Taskonomy: Disentangling Task Transfer Learning (2018, background): “[48], with representation size 500, is trained to do both relative camera pose prediction and point matching.”
Cognitive Mapping and Planning for Visual Navigation (2017, background): “Learning based approaches (Zamir et al. 2016; Hadsell et al. 2009) study the problem in isolation thus only learning generic task-independent maps.”
General-Purpose Deep Point Cloud Feature Extractor (2018, background): “There are other 3D datasets, such as those used for 3D estimation in cities [59, 36, 42].”
Self-supervised learning of a facial attribute embedding from video (2018, background): “Of more direct relevance to our training framework are self-supervised frameworks that use video data ([1, 8, 11, 16, 17, 21, 22, 30, 34, 44, 55, 56, 58, 61]).”
TorontoCity: Seeing the World with a Million Eyes (2016, methodology): “[37] utilizes 3D building models to generate correspondences from multiple streetview images.”
ACNe: Attentive Context Normalization for Robust Permutation-Equivariant Learning (2020, background): “This has changed over the past few years, with the arrival of deep learning and an abundance of new dense [57, 47, 60] and sparse [55, 12, 39, 59, 27] methods.”
Time-Contrastive Networks: Self-Supervised Learning from Multi-view Observation (2017, methodology): “[5, 33] use the spatial coherence in images as a self-supervision signal, [17] use motion cues to self-supervise a segmentation task, and [34] use camera pose estimation and feature matching.”
Comparative Evaluation of Hand-Crafted and Learned Local Features (2017, methodology): “In contrast, approaches that learn a new descriptor representation usually operate on raw image patches [3, 42, 43, 55].”
LF-Net: Learning Local Features from Images (2018, background): “This remains true even in competition with deep network alternatives, which typically involve dense matching [9, 43, 45, 52, 53] and tend to work best on narrow baselines, as they can suffer from occlusions, which local features are robust against.”
Learning to Find Good Correspondences (2017, methodology): “In contrast with traditional methods, many of these new techniques [32, 31, 37, 36] are dense, using the raw image as input.”
An Exploration of Embodied Visual Exploration (2020, unknown): “Next, we estimate the relative pose between the retrieved pairs and the query view to obtain pose estimates from each retrieval (Zamir et al. 2016).”
Zero-Shot Task Transfer (2019, background): “Table 4 shows the win rate of TTNet models on angular error with respect to state-of-the-art models: RANSAC [41], LRANSAC [18], G3D and Taskonomy [42].”
Mid-Level Visual Representations Improve Generalization and Sample Efficiency for Learning Active Tasks (2018, background): “…(Ofﬂine) Computer Vision encompasses a wide set of approaches—e.g. fully supervised learning [23], self-supervised learning [32, 35, 7, 56, 55, 33], unsupervised learning [9, 4, 7, 56, 8, 41]—adopted to solve various standard perception tasks, e.g. depth estimation [24], object detection…”