On the performance of ConvNet features for place recognition

By Niko Sünderhauf, Feras Dayoub, Sareh Abolahrari Shirazi, Ben Upcroft, Michael Milford. 2015

555 citations recorded by the source corpus.

After the incredible success of deep learning in the computer vision domain, there has been much interest in applying Convolutional Network (ConvNet) features in robotic fields such as visual navigation and SLAM. Unfortunately, there are fundamental differences and challenges involved. Computer vision datasets are very different in character to robotic camera data, real-time performance is essential, and performance priorities can be different. This paper comprehensively evaluates and compares the utility of three state-of-the-art ConvNets on the problems of particular relevance to navigation for robots; viewpoint-invariance and condition-invariance, and for the first time enables real-time place recognition performance using ConvNets with large maps by integrating a variety of existing (locality-sensitive hashing) and novel (semantic search space partitioning) optimization techniques. We present extensive experiments on four real world datasets cultivated to evaluate each of the specific challenges in place recognition. The results demonstrate that speed-ups of two orders of magnitude can be achieved with minimal accuracy degradation, enabling real-time performance. We confirm that networks trained for semantic place categorization also perform better at (specific) place recognition when faced with severe appearance changes and provide a reference for which networks and layers are optimal for different aspects of the place recognition problem.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Deep-learning in Mobile Robotics - from Perception to Control Systems: A Survey on Why and Why not (2016, background): “Application in place recognition for mobile robots were naturally motivated (Chen et al., 2014; S¨underhauf et al., 2015a).”
Visual Place Recognition: A Survey (2016, methodology): “viewpoint and carry more semantic information that can be used to partition the search space [207].”
Place Recognition with ConvNet Landmarks: Viewpoint-Robust, Condition-Robust, Training-Free (2015, methodology): “We clearly outperform the whole-image ConvNet-based method proposed by [41] and OpenFABMAP [14].”
UAV-YOLO: Small Object Detection on Unmanned Aerial Vehicle Perspective (2020, methodology): “Considering the high-efﬁciency, one-stage object detection attracts more attention, Liu et al. proposed the SSD method, which spreads out anchors of different scales to multilayers with ConvNet and enforces each layer to predict object at a certain scale.”
Deep learning a grasp function for grasping under gripper pose uncertainty (2016, background): “]. Consequently, these methods have also shown to be successful in robotics applications. Robot localisation is moving away from using hand-engineered features [29] and towards deep learning features [31], active recognition has achieved state-of-the-art performance by deep learning camera control [”
Learning Deployable Navigation Policies at Kilometer Scale from a Single Traversal (2018, unknown): “…network features like we use here have shown promising results toward robustly recognising places in the face of appearance and viewpoint change [36]; however, reducing perceptual aliasing and appearance differences is an active area of research in place recognition and SLAM [25] and many other…”
Fusion and binarization of CNN features for robust topological localization across seasons (2016, methodology): “CNN-based method defined in [26] (AlexNet conv3).”
Robust visual SLAM across seasons (2015, background): “[22] have reported Conv3 of Alexnet trained on ImageNet to behave more robust to seasonal changes on their datasets, we present a detailed comparison in Section IV.”
Benchmarking 6DOF Outdoor Visual Localization in Changing Conditions (2018, background): “Learning-based localization has been proposed to solve both loop-closure detection [17, 45, 64, 66] and pose estimation [19, 31, 74].”
Patch-NetVLAD: Multi-Scale Fusion of Locally-Global Descriptors for Place Recognition (2021, background): “Place matches obtained through global descriptor matching are often re-ranked using sequential information [24, 82, 46], query expansion [28, 13], geometric verification [38, 25, 49] and feature fusion [80, 83].”
Robotic grasp detection using deep convolutional neural networks (2016, background): “For example, robot localization is moving from using hand-engineered features [13] to deep learning features [14], deep reinforcement learning is being used for end-to-end training for robotic arm control [15], multi-view object recognition has achieved state-of-the-art performance by deep learning ”
Scan Context++: Structural Place Recognition Robust to Rotation and Lateral Variations in Urban Environments (2021, background): “, a fixed-size vector [16, 17] or a downsized image [5]) without maintaining a set of local keypoint descriptors.”
PoseNet: A Convolutional Network for Real-Time 6-DOF Camera Relocalization (2015, methodology): “Convnets have also been used to classify a scene into one of several location labels [23].”
Object identification from few examples by improving the invariance of a Deep Convolutional Neural Network (2016, background): “Indeed, it has been observed that training a standard classifier such as a Support Vector Machine (SVM) or a Regularized Least Squares Classifier (RLS) [16] on such features consistently outperforms more sophisticated competitors on a variety of challenging datasets [17], [3], [18], [19], [20], [10]”
Deep learning features at scale for visual place recognition (2017, background): “[8] was the first work to introduce a CNN-based place recognition system and [9] provided a thorough investigation of the utility and viewpoint-invariant properties of deep learnt features for place recognition.”
Training recurrent networks to generate hypotheses about how the brain solves hard navigation problems (2016, background): “SLAM algorithms are sometimes augmented by feedforward convolutional networks to assist in speciﬁc tasks like place recognition (see e.g. [29]) from camera images, but the geometric calculations and parameters at the core of SLAM algorithms – including noise models and continuity and smoothness…”