Neighbourhood Consensus Networks

By Ignacio Rocco, Mircea Cimpoi, Relja Arandjelovi'c, Akihiko Torii, Tomás Pajdla, Josef Sivic. 2018

445 citations recorded by the source corpus.

We address the problem of finding reliable dense correspondences between a pair of images. This is a challenging task due to strong appearance differences between the corresponding scene elements and ambiguities generated by repetitive patterns. The contributions of this work are threefold. First, inspired by the classic idea of disambiguating feature matches using semi-local constraints, we develop an end-to-end trainable convolutional neural network architecture that identifies sets of spatially consistent matches by analyzing neighbourhood consensus patterns in the 4D space of all possible correspondences between a pair of images without the need for a global geometric model. Second, we demonstrate that the model can be trained effectively from weak supervision in the form of matching and non-matching image pairs without the need for costly manual annotation of point to point correspondences. Third, we show the proposed neighbourhood consensus network can be applied to a range of matching tasks including both category- and instance-level matching, obtaining the state-of-the-art results on the PF Pascal dataset and the InLoc indoor visual localization benchmark.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Graph Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Graph Learning

Citations in the corpus

CrossTransformers: spatially-aware few-shot transfer (2020, background): “CrossTransformers establish soft correspondences between pixels in the query- and support-set images; such dense pairwise interactions [92] have recently proved useful for generative networks [99], semantic matching [66] and tracking [90].”
Temporal Cycle-Consistency Learning (2019, background): “This formulation has recently found application in metric learning for few-shot learning [28, 31, 40].”
COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning (2020, background): “As the sentences of a paragraph have a temporal order, given a sentence embedding δi on this sequence, we first find its soft nearest neighbor [36, 37, 38]”
Geometric Transformer for Fast and Robust Point Cloud Registration (2022, methodology): “Besides our powerful hybrid features, we also perform a dual-normalization operation [26,29] on S to further suppress ambiguous matches, leading to ¯ S with ¯ s i,j = s i,j We found that this suppression can effectively eliminate wrong matches.”
RoMa: Robust Dense Feature Matching (2023, background): “Robust Regression Losses: Robust loss functions provide a continuous transition between an inlier distribution (typically highly concentrated), and an outlier distribution (wide and flat).”
From Coarse to Fine: Robust Hierarchical Localization at Large Scale (2018, background): “Dense pixel-wise features naturally emerge from CNNs and provide a powerful representation used for image matching [10, 13, 35, 38] and localization [49, 42].”
Understanding the Limitations of CNN-Based Absolute Camera Pose Regression (2019, background): “Local features that are more robust to large viewpoint changes are an active field of research [51, 55] and structure-based methods will automatically benefit from progress in this field.”
COTR: Correspondence Transformer for Matching Across Images (2021, background): “Neighbourhood Consensus Networks [50] explored 4D correlations – while powerful, this limits the image size they can tackle.”
Hypercorrelation Squeeze for Few-Shot Segmenation (2021, methodology): “Figure 4: 4D convolution (left) and weights of 4D kernel [58, 81] (middle) and center-pivot 4D kernel (right).”
AiATrack: Attention in Attention for Transformer Visual Tracking (2022, methodology): “Several efforts have been made to recalibrate the raw correlations by processing them as features through hand-crafted algorithms [49,5] or learnable blocks [48,43,36,4,47,9].”
GAN-Supervised Dense Visual Alignment (2021, background): “[71] Ignacio Rocco, Mircea Cimpoi, Relja Arandjelović, Akihiko Torii, Tomas Pajdla, and Josef Sivic.”
ASpanFormer: Detector-Free Image Matching with Adaptive Span Transformer (2022, methodology): “Concurrent with detector-based matching, another line of works focus on generating correspondences directly from raw images [13,14,15,16,17,18,19,20,21], ar X iv :2 20 8.”
SuperGlue: Learning Feature Matching With Graph Neural Networks (2019, background): “Additionally, the optimal transport formulation enforces reciprocity of the matches, like the mutual check, but in a soft manner, similar to [43], thus embedding it into the training process.”
InLoc: Indoor Visual Localization with Dense Matching and View Synthesis (2019, background): “These matches can be used as a part of our localization pipeline, resulting in comparable results (InLoc+ [63] in Table 3).”
Image Matching from Handcrafted to Deep Features: A Survey (2020, methodology): “In addition, many methods integrate feature detection into the entire matching pipeline by jointly training with feature description and matching (Yi et al. 2016; DeTone et al. 2018; Ono et al. 2018; Shen et al. 2019; Dusmanu et al. 2019; Choy et al. 2016; Rocco et al. 2018; Dusmanu et al. 2019; Rev”
Visual Camera Re-Localization From RGB and RGB-D Images Using DSAC (2020, methodology): “Alternatively to making RANSAC differentiable, some authors propose to replace RANSAC by a neural network [60], [61], [62], [63], [64].”
LoFTR: Detector-Free Local Feature Matching with Transformers (2021, methodology): “NCNet [34] proposed a different approach by directly learning the dense correspondences in an end-toend manner.”