LIFT: Learned Invariant Feature Transform

By Kwang Moo Yi, Eduard Trulls, Vincent Lepetit, Pascal Fua. 2016

1,359 citations recorded by the source corpus.

We introduce a novel Deep Network architecture that implements the full feature point handling pipeline, that is, detection, orientation estimation, and feature description. While previous works have successfully tackled each one of these problems individually, we show how to learn to do all three in a unified manner while preserving end-to-end differentiability. We then demonstrate that our Deep pipeline outperforms state-of-the-art methods on a number of benchmark datasets, without the need of retraining.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning Spread-Out Local Feature Descriptors (2017, methodology): “The state-of-the-art descriptor learning methods are based on neural networks [1, 8, 19, 26].”
2D/3D Pose Estimation and Action Recognition Using Multitask Deep Learning (2018, methodology): “An intuitive graphical explanation of the Soft-argmax layer is shown in Figure 3.”
Matterport3D: Learning from RGB-D Data in Indoor Environments (2017, background): “With the recent success of neural networks, several works have begun to explore the use of deep learning techniques for training state-of-the-art keypoint descriptors that can facilitate robust matching between keypoints and their local image features [45, 34, 16].”
SIFT Meets CNN: A Decade Survey of Instance Retrieval (2016, methodology): “[154] further propose an end-to-end learning method integrating region detector, orientation estimator and feature descriptor in a single pipeline.”
H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions (2019, methodology): “In principle, it is straightforward to merge and train the single-image and temporal models jointly with a softargmax operation [5, 71].”
Key.Net: Keypoint Detection by Handcrafted and Learned CNN Filters (2019, methodology): “Furthermore, other CNNs also studied to perform tasks beyond detection or matching.”
UberNet: Training a Universal Convolutional Neural Network for Low-, Mid-, and High-Level Vision Using Diverse Datasets and Limited Memory (2016, background): “…only considering works that apply to a single, static image we can indicatively list successes of CNNs in superresolution [24], colorization [55], boundary detection [6,28,49,107], symmetry detection [91], interest point detection [109], image descriptors [35, 92, 110], surface Input Boundaries”
Unifying Deep Local and Global Features for Image Search (2020, background): “Recently, several deep learning-based local features have been proposed [63,39,33,42,34,49,14,6,29].”
PPF-FoldNet: Unsupervised Learning of Rotation Invariant 3D Local Descriptors (2018, background): “Already in 2D, learned descriptors significantly outperform their engineered counterparts [49, 28].”
3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions (2016, background): “This strategy of balancing positives and negatives has shown to be effective for efficiently learning discriminative descriptors [17, 32, 42].”
Learning to Estimate 3D Human Pose and Shape from a Single Color Image (2018, methodology): “To get the pixel position of the joints, we use an argsoftmax operation [11] on the heatmaps.”
Working hard to know your neighbor's margins: Local descriptor learning loss (2017, methodology): “As a ﬁrst step towards end-to-end learning, hand-crafted descriptors like SIFT [9, 10] or detectors [9, 11, 12] have been replace with learned ones, e.g., LIFT [13], MatchNet [14] and DeepCom-pare [15].”
Patch-NetVLAD: Multi-Scale Fusion of Locally-Global Descriptors for Place Recognition (2021, methodology): “Traditional handcrafted local feature methods such as SIFT [44], SURF [6] and ORB [60], and more recent deep-learned local features like LIFT [84], DeLF [52], SuperPoint [18] and D2Net [20], have been extensively employed for VPR [52, 9, 17], visual SLAM [50] and 6-DoF localization [63, 77, 20, 61].”
LF-Net: Learning Local Features from Images (2018, methodology): “Out of these baselines, our approach outperforms the closest competitor, LIFT, by 45% relative.”
Understanding the Limitations of CNN-Based Absolute Camera Pose Regression (2019, background): “, local features [63,79], outlier filtering [49,70], or scene coordinate regression [10,46], these approaches aim to learn the full localization pipeline.”
Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Pass (2025, neutral): “For example, [14, 19, 25, 42, 51, 68] improve feature extraction and correspondences, [27, 50, 59, 71] learn to estimate camera pose, and [52] introduce a bundle adjustment layer.”
SSD-6D: Making RGB-Based 3D Detection and 6D Pose Estimation Great Again (2017, methodology): “Rotation-invariant methods are based on local keypoints in either 2D [32] or 3D [7, 3, 30] by explicitly computing or voting for an orientation or a local reference frame, but they fail for objects of poor geometry or texture.”