SIFT Flow: Dense Correspondence across Scenes and Its Applications

By Ce Liu, Jenny Yuen, Antonio Torralba. 2011

1,784 citations recorded by the source corpus.

While image alignment has been studied in different areas of computer vision for decades, aligning images depicting different scenes remains a challenging problem. Analogous to optical flow, where an image is aligned to its temporally adjacent frame, we propose SIFT flow, a method to align an image to its nearest neighbors in a large image corpus containing a variety of scenes. The SIFT flow algorithm consists of matching densely sampled, pixelwise SIFT features between two images while preserving spatial discontinuities. The SIFT features allow robust matching across different scene/object appearances, whereas the discontinuity-preserving spatial model allows matching of objects located at different parts of the scene. Experiments show that the proposed approach robustly aligns complex scene pairs containing significant spatial differences. Based on SIFT flow, we propose an alignment-based large database framework for image analysis and synthesis, where image information is transferred from the nearest neighbors to a query image according to the dense scene correspondence. This framework is demonstrated through concrete applications such as motion field prediction from a single image, motion synthesis via object transfer, satellite image registration, and face recognition.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Panoptic Segmentation (2018, methodology): “, fully convolutional nets [30] were developed using [26, 9]).”
Generative Visual Manipulation on the Natural Image Manifold (2016, methodology): “Additional evaluation: In the supplemental material, we evaluate our motion+color flow method, and compare our results against popular alignment methods that are designed to handle large displacement between two images [44,45].”
Efficient Piecewise Training of Deep Structured Models for Semantic Segmentation (2015, neutral): “[28] 76.”
RefineNet: Multi-path Refinement Networks for High-Resolution Semantic Segmentation (2016, neutral): “[32] RGB-D − 10.”
An Uncertain Future: Forecasting from Static Images Using Variational Autoencoders (2016, methodology): “An improvement to this approach used dense-SIFT correspondence to align the matched objects in two images [12].”
Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks (2016, background): “[2015], Mathieu et al. [2016], user-supplied action labels Oh et al.”
Newtonian Image Understanding: Unfolding the Dynamics of Objects in Static Images (2015, background): “Data-driven approaches have been proposed in [37, 25] to predict motion field in a single image.”
Indoor Semantic Segmentation using depth information (2013, background): “Most datasets employed for semantic image segmentation [11, 17] present the objects centered into the images, under nice lightening conditions.”
Dense Optical Flow Prediction from a Static Image (2015, background): “For example, [28] simply retrieves videos visually similar to the static scene, allowing a warping [16] of the matched action into the scene.”
Deeper Depth Prediction with Fully Convolutional Residual Networks (2016, methodology): “A second cluster of related work comprises nonparametric approaches for depth transfer [10, 13, 18, 20], which typically perform feature-based matching (e.”
Unsupervised object discovery and localization in the wild: Part-based matching with bottom-up region proposals (2015, background): “, the SIFT flow [32], based on generic local regions.”
Generic 3D Representation via Pose Estimation and Matching (2016, methodology): “Since the patches often show non-planarity, we found employing a nonrigid transformation (we used SIFT flow [7]) for registration of the rectified patches to”
An Unsupervised Learning Model for Deformable Medical Image Registration (2018, background): “Extensions that better handle large displacements or dramatic changes in appearance include feature-based matching [9, 28] and nearest neighbor fields [10].”
Jump (2016, methodology): “…with classic methods that seek smooth solutions that satisfy brightness constancy assumptions [Horn and Schunk 1981; Lucas and Kanade 1981], to more recent approaches that use feature descriptors to address large displacements [Brox and Malik 2011] or appearance variation [Liu et al. 2011].”
Context Contrasted Feature and Gated Multi-scale Aggregation for Scene Segmentation (2018, neutral): “[32] - 10.”
Part-Aligned Bilinear Representations for Person Re-identification (2018, methodology): “Figure 4 and 5 In Figures 4 and 5, the normalized feature maps are visualized following SIFTFlow [62].”
Episodic CAMN: Contextual Attention-Based Memory Networks with Iterative Feedback for Scene Labeling (2017, neutral): “[36] 76.”
Learning Dense Correspondence via 3D-Guided Cycle Consistency (2016, methodology): “Cross-instance pairwise correspondence The classic SIFT Flow approach [26] proposes an energy minimization framework that computes dense correspondence between different scenes by matching SIFT features [28] regularized by smoothness and small displacement priors.”