Deep Tracking: Seeing Beyond Seeing Using Recurrent Neural Networks

By Peter Ondruska, Ingmar Posner. 2016

249 citations recorded by the source corpus.

This paper presents to the best of our knowledge the first end-to-end object tracking approach which directly maps from raw sensor input to object tracks in sensor space without requiring any feature engineering or system identification in the form of plant or sensor models. Specifically, our system accepts a stream of raw sensor data at one end and, in real-time, produces an estimate of the entire environment state at the output including even occluded objects. We achieve this by framing the problem as a deep learning task and exploit sequence models in the form of recurrent neural networks to learn a mapping from sensor measurements to object tracks. In particular, we propose a learning method based on a form of input dropout which allows learning in an unsupervised manner, only based on raw, occluded sensor data without access to ground-truth annotations. We demonstrate our approach using a synthetic dataset designed to mimic the task of tracking objects in 2D laser data — as commonly encountered in robotics applications — and show that it learns to track many dynamic objects despite occlusions and the presence of sensor noise.

Subfields: Aerospace Engineering, Artificial Intelligence, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Artificial Intelligence

In Control and Systems Engineering

Citations in the corpus

Deep Reinforcement Learning framework for Autonomous Driving (2017, methodology): “More recently, RNNs have also been used to improve object tracking performance in the DeepTracking model [13].”
Taking Visual Motion Prediction To New Heightfields (2017, unknown): “Several authors focused on learning to perform plausible physical predictions, for example to generate realistic future frames in a video Tompson et al. (2016); Ladický et al.”
End-to-End Deep Reinforcement Learning for Lane Keeping Assist (2016, methodology): “In [14] RNN was successfully applied for the task of end to end multi-object tracking.”
Applying Neural-Network-Based Machine Learning to Additive Manufacturing: Current Applications, Challenges, and Future Perspectives (2019, background): “For example, NNs are the main stimulating force in these areas: computer vision [13], voice recognition [14], natural language processing [15], and autonomous driving [16].”
Online Multi-Target Tracking Using Recurrent Neural Networks (2016, background): “Their follow-up work introduces real-world measurements and multi-class scenarios (Ondruska et al. 2016), however, in both cases, tracking is formulated as estimating the world occupancy, without explicit data association.”
From perception to decision: A data-driven approach to end-to-end motion planning for autonomous ground robots (2016, unknown): “[7] presented an end-to-end application of neural networks for dynamic object tracking using laser data.”
Probabilistic vehicle trajectory prediction over occupancy grid map via recurrent neural network (2017, result): “Recently, we found that similar LSTM-based trajectory analysis has been proposed independently in [12], [13].”
Distributed multi-robot collision avoidance via deep reinforcement learning for navigation in complex scenarios (2020, background): “Ondruska et al. (2016); Ondruska and Posner (2016) combined recurrent neural networks with convolutional operations to learn a direct mapping from raw 2D laser data to the unoccluded state of the entire scene; the mapping is then used for object tracking.”
Learning A Physical Long-term Predictor (2017, background): “The work of (Ondruska & Posner, 2016) also shows that an internal hidden state can be propagated through time using a simple deep recurrent architecture.”
Deep-Learned Collision Avoidance Policy for Distributed Multiagent Navigation (2016, methodology): “icies mapping raw image observations to torques at the robots motors. Deep neural networks have also been combined with model predictive control to control robotic systems [22], [23]. Ondruska et al. [24], [25] combined recurrent neural networks with convolutional operations to learn a mapping direc”
Sequence-to-Sequence Prediction of Vehicle Trajectory via LSTM Encoder-Decoder Architecture (2018, methodology): “In [12], the LSTM is used to track the position of the object based on the ranging sensor measurements.”
Auto-Encoding Sequential Monte Carlo (2017, methodology): “The dataset is inspired by (Ondrúška & Posner, 2016), however with the crucial difference that the movement of the agent is stochastic .”
End-to-end Recurrent Multi-Object Tracking and Trajectory Prediction with Relational Reasoning (2019, methodology): “Some approaches are targeted towards robustness across domains, for example by using a categoryagnostic object detector and performing classification only in a post-processing step [18, 19].”
IONet: Learning to Cure the Curse of Drift in Inertial Odometry (2018, background): “Sequential Deep Learning: Deep learning approaches have recently shown excellent performance in handling sequential data, such as speech recognition (Graves and Jaitly 2014), machine translation (Dai and Le 2015), visual tracking (Ondruska and Posner 2016) and video description (Donahue et al.”
Predictive-State Decoders: Encoding the Future into Recurrent Networks (2017, background): “For example, in a probabilistic filtering problem, the objective may be to minimize the negative log-likelihood of the observations [4, 52] or the prediction of the next observation [34].”
Backprop KF: Learning Discriminative Deterministic State Estimators (2016, background): “The use of recurrent neural networks (RNNs) for state estimation has been explored in several prior works [24, 4, 23, 19], but has generally been limited to simple tasks without complex sensory inputs such as images.”