COTR: Correspondence Transformer for Matching Across Images

By Wei Jiang, Eduard Trulls, Jan Hendrik Hosang, Andrea Tagliasacchi, Kwang Moo Yi. IEEE International Conference on Computer Vision, 2021

326 citations recorded by the source corpus.

We propose a novel framework for finding correspondences in images based on a deep neural network that, given two images and a query point in one of them, finds its correspondence in the other. By doing so, one has the option to query only the points of interest and retrieve sparse correspondences, or to query all points in an image and obtain dense mappings. Importantly, in order to capture both local and global priors, and to let our model relate between image regions using the most relevant among said priors, we realize our network using a transformer. At inference time, we apply our correspondence network by recursively zooming in around the estimates, yielding a multiscale pipeline able to provide highly-accurate correspondences. Our method significantly outperforms the state of the art on both sparse and dense correspondence problems on multiple datasets and tasks, ranging from wide-baseline stereo to optical flow, without any retraining for a specific dataset. We commit to releasing data, code, and all the tools necessary to train from scratch and ensure reproducibility.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/1536febbc3e15d51642991797f6ce624838a468e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives (2023, unknown): “Related tasks are image-level sparse correspondence given query points (instead of object masks) [65] and image-level object co-segmentation [166] for jointly segmenting semantically similar objects.”
Grounding Image Matching in 3D with MASt3R (2024, methodology): “In contrast to keypoint-based approaches, semi-dense [11,16,43,46,82,85] and dense approaches [27,28,29,58,92,93,94,122] offer a different paradigm for establishing image correspondences, considering all possible pixel associations.”
GMFlow: Learning Optical Flow via Global Matching (2021, methodology): “Although COTR in principle can also predict dense correspondences by querying every pixel location, the inference speed will be significantly slower.”
TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement (2023, methodology): “Keypoint Correspondence methods aim at sparse keypoint matching given image pairs, from hand-defined discriminative feature descriptors [3, 32, 33, 49] to deep learning based methods [11, 25, 35, 40].”
FlowFormer: A Transformer Architecture for Optical Flow (2022, methodology): “Visual correspondence tasks [45,19,7,27,53] is a main stream in computer vision.”
A Survey of Visual Transformers (2021, methodology): “TransFuser [128], COTR [129], and mmFormer [133] deploy a hybrid model.”
Deep ViT Features as Dense Visual Descriptors (2021, methodology): “Recent works employ transformers for dense correspondence in intra-class pairs [8, 47, 22].”
TAP-Vid: A Benchmark for Tracking Any Point in a Video (2022, unknown): “COTR [30]: This method is intended to be used with larger baselines, so to propagate a query point, we apply this algorithm with the query frame paired with every other frame.”
ASpanFormer: Detector-Free Image Matching with Adaptive Span Transformer (2022, methodology): “As validated in Transformer networks [13,30,14], positional encoding is critical in maintaining spatial information for the flattened tokens.”