Learning to Track at 100 FPS with Deep Regression Networks

By David Held, Sebastian Thrun, Silvio Savarese. 2016

1,257 citations recorded by the source corpus.

Machine learning techniques are often used in computer vision due to their ability to leverage large amounts of training data to improve performance. Unfortunately, most generic object trackers are still trained from scratch online and do not benefit from the large number of videos that are readily available for offline training. We propose a method for offline training of neural networks that can track novel objects at test-time at 100 fps. Our tracker is significantly faster than previous methods that use neural networks for tracking, which are typically very slow to run and not practical for real-time applications. Our tracker uses a simple feed-forward network with no online training required. The tracker learns a generic relationship between object motion and appearance and can be used to track novel objects that do not appear in the training set. We test our network on a standard tracking benchmark to demonstrate our tracker's state-of-the-art performance. Further, our performance improves as we add more videos to our offline training set. To the best of our knowledge, our tracker is the first neural-network tracker that learns to track generic objects at 100 fps.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

A Survey of Autonomous Driving: Common Practices and Emerging Technologies (2019, methodology): “Tracking in monocular images was achieved in real-time through a CNN-based method [191], [192].”
Fully-Convolutional Siamese Networks for Object Tracking (2016, methodology): “[28] introduce GOTURN, in which a conv-net is trained to regress directly from two images to the location in the second image of the object shown in the first image.”
Learning Video Object Segmentation from Static Images (2016, methodology): “GOTURN [18] proposes to train ofﬂine a convnet so as to directly regress the bounding box in the current frame based on the object position and appearance in the previous frame.”
Risky Action Recognition in Lane Change Video Clips using Deep Spatiotemporal Networks with Segmentation Mask Transfer (2019, background): “Developments in machine learning, particularly through deep convolutional neural networks (CNNs), significantly increased object detection capabilities [3] and made reliable object tracking achievable [4].”
Quasi-Dense Similarity Learning for Multiple Object Tracking (2020, methodology): “Some methods use motion priors, such as Kalman Filter [4, 52], optical flow [48], and displacement regression [10, 16], to ensure accurate distance estimations.”
Action-Decision Networks for Visual Tracking with Deep Reinforcement Learning (2017, methodology): “Recently proposed methods [31, 24, 12] overcome the gap by training their network with a large amount of tracking video datasets [40, 20, 29].”
Learning Spatio-Temporal Transformer for Visual Tracking (2021, background): “There are some trackers [18, 21] attempting to simplify the tracking pipeline, but their performances still lag far behind that of state-of-the-art trackers.”
Deep Learning for Visual Tracking: A Comprehensive Survey (2019, methodology): “These methods include the HCFT [59], DeepSRDCF [60], FCNT [61], CNNSVM [62], DPST [63], CCOT [64], GOTURN [65], SiamFC [66], SINT [67], MDNet [68], HDT [69], STCT [70], RPNT [71], DeepTrack [72], CNT [73], CF-CNN [74], TCNN [75], RDLT [76], PTAV [77], [78], CREST [79], UCT/UCTLite [80], DSiam/DSiamM”
Meta-Tracker: Fast and Robust Online Adaptation for Visual Object Trackers (2018, background): “Ofﬂine-only trained trackers with deep learning have also been suggested, with promising results and high speed, but accuracy has fallen short compared to state-of-the-art online adaptive trackers [6, 20, 48].”
Unveiling the Power of Deep Tracking (2018, background): “Another approach is to directly regress the relative target location given an input patch [14, 33].”
Detect to Track and Track to Detect (2017, methodology): “…the images from the DET training set, this has an additional beneﬁcial effect of letting our model prefer small motions over large ones ( e.g . the tracker in [13] samples motion augmentation from a Laplacian distribution with zero mean to bias a regression tracker on small displacements).”
Learning Discriminative Model Prediction for Tracking (2019, neutral): “MDNet CF2 ECO CCOT GOTURN SiamFC SiamFCv2 ATOM DiMP-18 DiMP-50 [29] [24] [7] [9] [14] [2] [40] [6]”
The RobotriX: An Extremely Photorealistic and Very-Large-Scale Indoor Dataset of Sequences with Robot Trajectories and Interactions (2018, background): “Various works have highlighted the importance of high-resolution and high frame-rate data for various computer vision applications [1][2].”
SiamRPN++: Evolution of Siamese Visual Tracking With Very Deep Networks (2018, methodology): “Additionally, these two branches can remain fixed during the tracking phase [42, 1, 16, 45, 25, 54] or updated online to adapt the appearance changes of the target [44, 43, 13].”
CoTracker: It is Better to Track Together (2023, background): “However, this approximation is crude as points are often strongly correlated ( e.g ., when they belong to the same object).”
The Unmanned Aerial Vehicle Benchmark: Object Detection and Tracking (2018, background): “…features (KCF [21], Staple-CA [32], and SRDCFdecon [11]); II) CF trackers with deep features (ECO [9], C-COT [12], HDT [36], CF2 [28], CFNet [43], and PTAV [17]); III) Deep trackers (MDNet [33], SiamFC [5], FCNT [44], SINT [42], MCPF [53], GOTURN [20], ADNet [52], CREST [41], and STCT [45]).”
End-to-End Representation Learning for Correlation Filter Based Tracking (2017, methodology): “Recent works have focused on learning deep embeddings that can be used as universal object descriptors [3, 12, 28, 17, 5].”
Need for Speed: A Benchmark for Higher Frame Rate Object Tracking (2017, methodology): “There are two high-speed deep trackers GOTURN [12] and SFC [2] that are able to run at 100 FPS and 75 FPS respectively on GPUs.”
StarNet: Targeted Computation for Object Detection in Point Clouds (2019, background): “Finally, we are particularly interested in studying how this system may be amenable to object tracking [3, 17, 16] as we suspect that because of the design, the computational demands may scale as the difference between successive time points as opposed to operating on the entirety of the scene [11].”