DeepVO: Towards end-to-end visual odometry with deep Recurrent Convolutional Neural Networks

By Sen Wang, R. Clark, Hongkai Wen, Agathoniki Trigoni. IEEE International Conference on Robotics and Automation, 2017

916 citations recorded by the source corpus.

This paper studies monocular visual odometry (VO) problem. Most of existing VO algorithms are developed under a standard pipeline including feature extraction, feature matching, motion estimation, local optimisation, etc. Although some of them have demonstrated superior performance, they usually need to be carefully designed and specifically fine-tuned to work well in different environments. Some prior knowledge is also required to recover an absolute scale for monocular VO. This paper presents a novel end-to-end framework for monocular VO by using deep Recurrent Convolutional Neural Networks (RCNNs). Since it is trained and deployed in an end-to-end manner, it infers poses directly from a sequence of raw RGB images (videos) without adopting any module in the conventional VO pipeline. Based on the RCNNs, it not only automatically learns effective feature representation for the VO problem through Convolutional Neural Networks, but also implicitly models sequential dynamics and relations using deep Recurrent Neural Networks. Extensive experiments on the KITTI VO dataset show competitive performance to state-of-the-art methods, verifying that the end-to-end Deep Learning technique can be a viable complement to the traditional VO systems.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Ocean Engineering

View Paper: https://www.semanticscholar.org/paper/60c67a6d529e8e6f55c239f5a16c80c6c4996d89

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Ocean Engineering

Citations in the corpus

DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras (2021, background): “Other work has tried to learn SLAM or VO systems end-to-end [59, 47, 53, 52, 46].”
NeurAll: Towards a Unified Visual Perception Model for Automated Driving (2019, neutral): “[19] use a combination of”
DeepV2D: Video to Depth with Differentiable Structure from Motion (2018, background): “Kendall et. al [20, 19] focused on the problem of camera localization, while other work [49, 42, 44, 48, 41] aim to estimate camera motion between a pair of frames.”
FutureMapping: The Computational Structure of Spatial AI Systems (2018, methodology): “[55] on estimating incremental visual odometry from video using an RNN whose training was via a supervised known pose signal.”
UnDeepVO: Monocular Visual Odometry Through Unsupervised Deep Learning (2017, methodology): “Wang et al. [11] then presented a Recurrent Convolutional Neural Network (RCNN) based VO method resulting in a competitive performance against model-based VO methods.”
IONet: Learning to Cure the Curse of Drift in Inertial Odometry (2018, background): “Previous learningbased work has tackled localization problems, by employing visual odometry (Zhou et al. 2017; Wang et al. 2017; Clark et al. 2017a) and visual inertial odometry (Clark et al.”
NeurAll: Towards a Unified Model for Visual Perception in Automated Driving (2019, neutral): “[19] use a combination of”
Adaptive VIO: Deep Visual-Inertial Odometry with Online Continual Learning (2024, positive): “End-to-end learning-based VO utilizes pose and depth estimation networks to replace classic modules of track ing and mapping, trained in either a supervised [40, 43] or self-supervised [18, 45] manner.”
D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometry (2020, background): “End-to-end learned deep neural networks have been explored to directly predict the relative poses between images with supervised [70, 75, 85] or unsupervised learning [46, 73, 82, 86].”
A survey of uncertainty in deep neural networks (2021, background): “[207], [330], [331] developed deep learning based localization systems along with uncertainty estimates.”
Visual SLAM and Structure from Motion in Dynamic Environments (2018, background): “[171] propose “DeepVO” as an end-to-end learning framework capable of learning sequential motion dynamics from image sequences through a Recurrent Convolutional Neural Network (RCNN), a combination of CNN and Recurrent Neural Network (RNN).”
MapNet: An Allocentric Spatial Memory for Mapping Environments (2018, methodology): “We also reimplemented the state-of-the-art DeepVO method [24], which is similar but uses a FlowNet CNN [5].”
Global Pose Estimation with an Attention-Based Recurrent Network (2018, methodology): “The architecture of the Local Pose Estimation network is inspired by the architecture of Flownet [11] which predicts the optical flow between two frames.”
DeepIM: Deep Iterative Matching for 6D Pose Estimation (2018, methodology): “We tried using the VGG16 image classification network (Simonyan and Zisserman, 2014) as the backbone network, but the results were very poor, confirming the intuition that a representation related to optical flow is very useful for pose matching (Wang et al., 2017).”
Deep Virtual Stereo Odometry: Leveraging Deep Depth Prediction for Monocular Direct Sparse Odometry (2018, methodology): “We also compare DVSO with DeepVO [41], UnDeepVO [26] and SfMLearner [49] which are deep learning based visual odometry systems trained end-to-end on KITTI.”
Who Let the Dogs Out? Modeling Dog Behavior from Visual Data (2018, methodology): “[45] propose a method based on combination of CNNs and RNNs to perform ego-motion estimation for cars.”
Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction (2018, background): “Wang et al.[38] presented a recurrent ConvNet architecture for learning monocular odometry from video sequences.”
Deep Reinforcement Learning Robot for Search and Rescue Applications: Exploration in Unknown Cluttered Environments (2019, methodology): “Traditional machine learning and deep learning techniques have been applied to image classification [13]–[16], target object detection [17], [18], robot-team work division [18], and robot visual odometry [19], [20] problems.”
Distilling Knowledge From a Deep Pose Regressor Network (2019, methodology): “In particular, we employ DNN-based VO methods [33, 34, 37, 38, 26, 1] which replaces the conventional VO pipeline based on multiple-view geometry [15, 27] to DNN-based pipeline which automatically learns useful features for estimating 6 Degree-of-Freedom (DoF) camera poses.”
CodeSLAM - Learning a Compact, Optimisable Representation for Dense Visual SLAM (2018, background): “Some particularly eye-catching pieces of work over the past year have focused on supervised and self-supervised training of surprisingly capable networks which are able to estimate visual odometry, depth and other quantities from video [10, 29, 30, 4, 32, 31].”