Simple online and realtime tracking with a deep association metric

By Nicolai Wojke, Alex Bewley, Dietrich Paulus. 2017

4,744 citations recorded by the source corpus.

Simple Online and Realtime Tracking (SORT) is a pragmatic approach to multiple object tracking with a focus on simple, effective algorithms. In this paper, we integrate appearance information to improve the performance of SORT. Due to this extension we are able to track objects through longer periods of occlusions, effectively reducing the number of identity switches. In spirit of the original framework we place much of the computational complexity into an offline pre-training stage where we learn a deep association metric on a largescale person re-identification dataset. During online application, we establish measurement-to-track associations using nearest neighbor queries in visual appearance space. Experimental evaluation shows that our extensions reduce the number of identity switches by 45%, achieving overall competitive performance at high frame rates.

Subfields: Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Occluded Video Instance Segmentation: A Benchmark (2021, methodology): “Moreover, our methods also beat other representative methods by a larger margin, including DeepSORT [43], STEm-Seg [1], etc.”
End-to-End Video Instance Segmentation with Transformers (2020, neutral): “Method backbone FPS AP AP50 AP75 AR1 AR10 DeepSORT [28] ResNet-50 - 26.”
Video Instance Segmentation (2019, methodology): “Methods [26, 32, 28] under this setting usually take the “trackingby-detection” strategy.”
Deep OC-Sort: Multi-Pedestrian Tracking by Adaptive Re-Identification (2023, methodology): “We note that the adaptive way we incorporate visual appearance with the motion-based method is newly designed, instead of a straightforward adaptation of what DeepSORT [8] does upon SORT [9].”
Beyond Short-Term Snippet: Video Relation Detection With Spatio-Temporal Global Context (2020, methodology): “We choose deep sort [41] as our tracker which can integrate visual features as matching descriptors to improve tracking performance.”
Chained-Tracker: Chaining Paired Attentive Regression Results for End-to-End Joint Multiple-Object Detection and Tracking (2020, background): “0 DeepSORT [16] 61.”
Towards Real-Time Multi-Object Tracking (2019, methodology): “State-of-the-art online MOT methods under the private protocol are compared, including DeepSORT 2 [38], RAR16wVGG [10], TAP [49], CNNMTT [23] and POI [40].”
CityFlow: A City-Scale Benchmark for Multi-Target Multi-Camera Vehicle Tracking and Re-Identification (2019, methodology): “DeepSORT [52] is an online method that combines deep learning features with Kalman-filter-based tracking and the Hungarian algorithm for data association, achieving remarkable performance on the MOTChallenge MOT16 benchmark [31].”
Rethinking the Competition Between Detection and ReID in Multiobject Tracking (2020, methodology): “Therefore, some works [2]–[5] attempt to apply ReID network to yield more robust predictions.”
BoT-SORT: Robust Associations Multi-Pedestrian Tracking (2022, methodology): “In more recent trackers [48, 46, 59, 58, 12] the state vector has changed to an eight-tuple, x = [ x c , y c , a, h, ˙ x c , ˙ y c , ˙ a, ˙ h ] (cid:62) .”
Learning From Demonstration in the Wild (2018, methodology): “To validate the ViBe vision module, we manually label 43 trajectories from the dataset and then compare its performance against two baselines, a simple IOU [18] tracker and Deep SORT [13], [14], a state-of-the-art MOT pipeline.”
Memory Based Online Learning of Deep Representations from Video Streams (2017, background): “Few exceptions can operate online and use deep features [41, 42, 43, 44] but they still assume continuous motion and do not update the appearance.”
Deep learning in video multi-object tracking: A survey (2019, methodology): “Fu et al. [98] employed the DeepSORT feature extractor, and measured the correlation of features using a discriminative correlation filter.”
Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Tracking (2022, methodology): “DeepSORT [63] further introduces deep visual features [23, 51] into object association under the framework of SORT.”
Dual Embedding Learning for Video Instance Segmentation (2019, methodology): “Only with the instance-level tracking, DeepSORT [12] is used to link the detection results of Mask R-CNN, which could only reach mAP 25.7.”