Universal Correspondence Network

By Christopher Bongsoo Choy, JunYoung Gwak, Silvio Savarese, Manmohan Chandraker. Neural Information Processing Systems, 2016

379 citations recorded by the source corpus.

We present a deep learning framework for accurate visual correspondences and demonstrate its effectiveness for both geometric and semantic matching, spanning across rigid motions to intra-class shape or appearance variations. In contrast to previous CNN-based approaches that optimize a surrogate patch similarity objective, we use deep metric learning to directly learn a feature space that preserves either geometric or semantic similarity. Our fully convolutional architecture, along with a novel correspondence contrastive loss allows faster training by effective reuse of computations, accurate gradient computation through the use of thousands of examples per image pair and faster testing with $O(n)$ feed forward passes for $n$ keypoints, instead of $O(n^2)$ for typical patch similarity methods. We propose a convolutional spatial transformer to mimic patch normalization in traditional features like SIFT, which is shown to dramatically boost accuracy for semantic correspondences across intra-class shape variations. Extensive experiments on KITTI, PASCAL, and CUB-2011 datasets demonstrate the significant advantages of our features over prior works that use either hand-constructed or learned features.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/6b581c1ee47e215456a2908309606486d7a559cc

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

From Coarse to Fine: Robust Hierarchical Localization at Large Scale (2018, background): “Dense pixel-wise features naturally emerge from CNNs and provide a powerful representation used for image matching [10, 13, 35, 38] and localization [49, 42].”
ASLFeat: Learning Local Features of Accurate Shape and Localization (2020, methodology): “As an alternative, UCN [4] and LF-Net [27] takes images as input and performs STN [14] on intermediate features.”
Dense Contrastive Learning for Self-Supervised Visual Pre-Training (2020, background): “Recently, a variety of convolutional neural network based approaches are proposed to measure the similarity between patches across images, including both supervised [4, 21] and unsupervised ones [45, 15].”
AutoScaler: Scale-Attention Networks for Visual Correspondence (2016, methodology): “When the threshold α becomes large, our method ranks second among all the competing algorithm, following UCN [13].”
Visual SLAM for Automated Driving: Exploring the Applications of Deep Learning (2018, background): “For example, a universal correspondence network in [12] by making use of a spatial transformer to normalizer for affine transformations demonstrates state-of-the-art results in various datasets.”
Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction (2018, background): “LIFT [42] and MC-CNN [43] learn a similarity measure on small image patches while [40][2] learns fully convolutional features good for matching.”
Self-Supervised Visual Descriptor Learning for Dense Correspondence (2017, background): “[4], and Gadot and Wolf [7]) to identify dense correspondences in images”
Neighbourhood Consensus Networks (2018, neutral): “Finding matches consistent with a geometric model is typically performed in a separate post-processing stage [22, 17, 10, 3, 37, 36, 4, 43, 7, 26].”
D2-Net: A Trainable CNN for Joint Description and Detection of Local Features (2019, methodology): “An alternative to the detect-then-describe approach is to forego the detection stage and perform the description stage densely across the whole image [10, 15, 48, 52].”
Self-Supervised Learning of Geometrically Stable Features Through Probabilistic Introspection (2018, methodology): “To overcome this, deep features have been retrained for geometric tasks [4, 43, 9].”
Learning Metrics From Teachers: Compact Networks for Image Embedding (2019, background): “Furthermore, embedding networks are essential for computer vision, as evidenced by the large variety of tasks in which they are used, including feature-based object retrieval [9], face recognition [25], feature matching [6], domain adaptation [27], weakly supervised learning [36], ranking [35], or z”
SEGCloud: Semantic Segmentation of 3D Point Clouds (2017, methodology): “Within theCaf fe framework, we adapted the bilinear interpolation of [11] and implemented trilinear interpolation as a neural network layer.”
SuperPoint: Self-Supervised Interest Point Detection and Description (2017, methodology): “To output a dense map of L2normalized fixed length descriptors, we use a model similar to UCN [3] to first output a semi-dense grid of descriptors (e.g., one every 8 pixels).”
Deep Metric Learning via Facility Location (2016, background): “Learning to measure the similarity among arbitrary groups of data is of great practical importance, and can be used for a variety of tasks such as feature based retrieval [30], clustering [10], near duplicate detection [41], verification [3, 4], feature matching [6], domain adaptation [27], video ba”
R2D2: Repeatable and Reliable Detector and Descriptor (2019, methodology): “Ranking losses have thus been extensively and successfully used to train local descriptors ( e.g . triplet loss [6, 3, 21, 55, 29, 61]).”
Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation (2018, methodology): “For cross-instance semantic correspondence, [7] relies on human annotations, while [8] learns these unsupervised, as we do here.”
Image Matching from Handcrafted to Deep Features: A Survey (2020, methodology): “…jointly training with feature description and matching (Yi et al. 2016; DeTone et al. 2018; Ono et al. 2018; Shen et al. 2019; Dusmanu et al. 2019; Choy et al. 2016; Rocco et al. 2018; Dusmanu et al. 2019; Revaud et al. 2019), which can enhance the final matching performance and optimize the…”
LAF-Net: Locally Adaptive Fusion Networks for Stereo Confidence Estimation (2019, methodology): “To deal with locally-varying scale ﬁelds, we ﬁrst build a locally-varying sampling grid for N × N neighbors j ∈ N i indenpendently, and then warp the convolutional activation for each sampling grid as used in [4, 19].”