Convolutional Neural Network Architecture for Geometric Matching

By Ignacio Rocco, Relja Arandjelovi'c, Josef Sivic. 2017

586 citations recorded by the source corpus.

We address the problem of determining correspondences between two images in agreement with a geometric model such as an affine or thin-plate spline transformation, and estimating its parameters. The contributions of this work are three-fold. First, we propose a convolutional neural network architecture for geometric matching. The architecture is based on three main components that mimic the standard steps of feature extraction, matching and simultaneous inlier detection and model parameter estimation, while being trainable end-to-end. Second, we demonstrate that the network parameters can be trained from synthetically generated imagery without the need for manual annotation and that our matching layer significantly increases generalization capabilities to never seen before images. Finally, we show that the same model can perform both instance-level and category-level matching giving state-of-the-art results on the challenging Proposal Flow dataset.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Self-Supervised Learning of Pretext-Invariant Representations (2019, background): “Although such covariance is beneficial for tasks such as predicting 3D correspondences [33, 57, 65], it is undesirable for most semantic recognition tasks.”
Deformable Kernels: Adapting Effective Receptive Fields for Object Deformation (2019, unknown): “…free-form sampling in image space: Spatial Transformers (Jaderberg et al., 2015) learns 2D afﬁne transformations, Deep Geometric Matchers (Rocco et al., 2017) learns thin-plate spline transformations, Deformable Convolutions (Dai et al., 2017; Zhu et al., 2019) learns free-form…”
DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision (2021, methodology): “More recent methods have universally resorted to deep networks as powerful feature extractors [21–23].”
SuperPoint: Self-Supervised Interest Point Detection and Description (2017, background): “Also, Geometric Matching Networks [20] and Deep Image Homography Estimation [4] use a similar self-supervision strategy to create training data for estimating global transformations.”
Hypercorrelation Squeeze for Few-Shot Segmenation (2021, methodology): “Many methods [21, 22, 30, 38, 42, 44, 56, 58, 81] typically built upon convolutional features pretrained on classification task [9], showing they serve as good transferable representations.”
Image Matching from Handcrafted to Deep Features: A Survey (2020, methodology): “Similar methods regarding deep networks as a regressor can directly learn the parameter transform model from image pairs, such as Fundamental (Poursaeed et al. 2018), Homography (DeTone et al. 2016), and non-rigid deformation (Rocco et al. 2017).”
Self-Supervised Learning of Geometrically Stable Features Through Probabilistic Introspection (2018, methodology): “To avoid border artifacts, following [29], we mirror-pad each image enlarging its size by a factor of two while biasing the sampled transformations towards zooming into the padded image.”
Under review as a conference paper at ICLR 2020 completely free-form sampling in image space : Spatial Transformers (2019, background): “Another line of work on this topic learns to recompose data by either semi-parameterized or completely free-form sampling in image space: Spatial Transformers (Jaderberg et al., 2015) learns 2D affine transformations, Deep Geometric Matchers (Rocco et al., 2017) learns thin-plate spline transformati”
Learning Two-View Correspondences and Geometry Using Order-Aware Network (2019, methodology): “With the emergence of deep learning, many works attempted to employ learning-based methods to solve geometric matching tasks, including both dense methods [36, 45, 13, 30] and sparse methods [40, 23, 4, 3, 18, 17].”
Positional Encoding as Spatial Inductive Bias in GANs (2020, background): “It is widely adopted for differentiable image warping [38, 44, 49, 51, 50, 48] and aligning pixels among different spaces [10, 17, 27, 40, 41, 53].”
VideoMatch: Matching based Video Object Segmentation (2018, methodology): “Correlation between the extracted feature maps is typically computed to find correspondences [45], to estimate optical flow fields [10] and geometric transformations [46].”
GAN-Supervised Dense Visual Alignment (2021, background): “[69] Ignacio Rocco, Relja Arandjelovic, and Josef Sivic.”
Unsupervised Discovery of Object Landmarks as Structural Representations (2018, background): “[50] proposed to discover TPS control points to match pairs of object images densely.”
ClothFlow: A Flow-Based Model for Clothed Person Generation (2019, methodology): “Our flow estimation network predicts more accurate deformations than existing methods [39, 4, 33] which compute a thin plane spline (TPS) transformation.”
A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence (2023, methodology): “Conventional methods consist of three steps [64]: feature extraction, cost volume construction, and displacement field [60–63] or parameterized transformation regression [25, 28, 46, 47, 54].”
Under review as a conference paper at ICLR 2020 completely free-form sampling in image space : Spatial Transformers (2019, background): “…free-form sampling in image space: Spatial Transformers (Jaderberg et al., 2015) learns 2D affine transformations, Deep Geometric Matchers (Rocco et al., 2017) learns thin-plate spline transformations, Deformable Convolutions (Dai et al., 2017; Zhu et al., 2019) learns free-form…”
D2-Net: A Trainable CNN for Joint Description and Detection of Local Features (2019, background): “As in other previous work [37,43,58], the most straightforward interpretation of the 3D tensor F is as a dense set of descriptor vectors d:”
Self-supervised Video Representation Learning for Correspondence Flow (2019, unknown): “[34, 35], propose to train deep convolutional neural networks by learning the artificial transformations between a pair of images.”