Image-Based Localization Using LSTMs for Structured Feature Correlation

By Florian Walch, Caner Hazirbas, L. Leal-Taix'e, Torsten Sattler, Sebastian Hilsenbeck, Daniel Cremers. 2016

545 citations recorded by the source corpus.

In this work we propose a new CNN+LSTM architecture for camera pose regression for indoor and outdoor scenes. CNNs allow us to learn suitable feature representations for localization that are robust against motion blur and illumination changes. We make use of LSTM units on the CNN output, which play the role of a structured dimensionality reduction on the feature vector, leading to drastic improvements in localization performance. We provide extensive quantitative comparison of CNN-based and SIFT-based localization methods, showing the weaknesses and strengths of each. Furthermore, we present a new large-scale indoor dataset with accurate ground truth from a laser scanner. Experimental results on both indoor and outdoor public datasets show our method outperforms existing deep architectures, and can localize images in hard conditions, e.g., in the presence of mostly textureless surfaces, where classic SIFT-based methods fail.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Geology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Geology

Citations in the corpus

CPlaNet: Enhancing Image Geolocalization by Combinatorial Partitioning of Maps (2018, methodology): “The accuracy of PoseNet is improved by introducing an intermediate LSTM layer for dimensionality reduction [27].”
Benchmarking 6DOF Outdoor Visual Localization in Changing Conditions (2018, background): “They learn features with stable appearance over time [17, 47, 49], train classifiers for place recognition [12, 28, 37, 76], and train CNNs to regress 2D-3D matches [8, 9, 60] or camera poses [19, 31, 74].”
Back to the Feature: Learning Robust Camera Localization from Pixels to Pose (2021, background): “An alternative is to train a convolutional neural network (CNN) to regress geometric quantities such as camera poses [5, 24, 35, 37, 42, 90, 97] or the 3D scene coordinate corresponding to each pixel [9–11, 13, 16, 17, 45, 80, 93].”
Meta Module Generation for Fast Few-Shot Incremental Learning (2019, methodology): “[34] has reported that the LSTM network is a powerful tool for feature correlation.”
A survey of deep learning techniques for autonomous driving (2019, methodology): “Neural networks, such as PoseNet (Kendall, Grimes, & Cipolla, 2015), VLocNet++ (Radwan, Valada, & Burgard, 2018), or the approaches introduced in Walch et al. (2017), Melekhov, Ylioinas, Kannala, and Rahtu (2017), Laskar, Melekhov, Kalia, and Kannala (2017), Brachmann and Rother (2018), or Sarlin,…”
The ApolloScape Open Dataset for Autonomous Driving and Its Application (2018, background): “LSTM-PoseNet [49] further captures a global spatial context after CNN features.”
Geometry-Aware Learning of Maps for Camera Localization (2017, methodology): “Our work is built upon prior works in DNN-based pose estimation methods [11, 30, 31, 32, 40, 55], which regress 6-DoF camera pose from an input RGB image with a DNN.”
Event-Based Vision Meets Deep Learning on Steering Prediction for Self-Driving Cars (2018, background): “perform well on many applications in the field of motion estimation [7, 8, 9].”
L3-Net: Towards Learning Based LiDAR Localization for Autonomous Driving (2019, background): “PoseNet [15] and its variants [14, 25, 31] attempt to solve the visual re-localization problem, in which an accurate solution is not the goal.”
Understanding the Limitations of CNN-Based Absolute Camera Pose Regression (2019, methodology): “Yet, they are also significantly less accurate than structure-based methods [10, 63, 76].”