Towards Real-Time Multi-Object Tracking

By Zhongdao Wang, Liang Zheng, Yixuan Liu, Shengjin Wang. European Conference on Computer Vision, 2019

1,115 citations recorded by the source corpus.

Modern multiple object tracking (MOT) systems usually follow the tracking-by-detection paradigm. It has 1) a detection model for target localization and 2) an appearance embedding model for data association. Having the two models separately executed might lead to efficiency problems, as the running time is simply a sum of the two steps without investigating potential structures that can be shared between them. Existing research efforts on real-time MOT usually focus on the association step, so they are essentially real-time association methods but not real-time MOT system. In this paper, we propose an MOT system that allows target detection and appearance embedding to be learned in a shared model. Specifically, we incorporate the appearance embedding model into a single-shot detector, such that the model can simultaneously output detections and the corresponding embeddings. As such, the system is formulated as a multi-task learning problem: there are multiple objectives, i.e., anchor classification, bounding box regression, and embedding learning; and the individual losses are automatically weighted. To our knowledge, this work reports the first (near) real-time MOT system, with a running speed of 18.8 to 24.1 FPS depending on the input resolution. Meanwhile, its tracking accuracy is comparable to the state-of-the-art trackers embodying separate detection and embedding (SDE) learning (64.4% MOTA v.s. 66.1% MOTA on MOT-16 challenge). The code and models are available at this https URL.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Safety, Risk, Reliability and Quality

View Paper: https://www.semanticscholar.org/paper/a62e40be6c2078ace369ce58801b3d5d1dd1a351

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Safety, Risk, Reliability and Quality

Citations in the corpus

DanceTrack: Multi-Object Tracking in Uniform Appearance and Diverse Motion (2021, methodology): “For example, JDE [31] and FairMOT [39] learn object localization and appearance embedding using a shared backbone for better appearance representation.”
FairMOT: On the Fairness of Detection and Re-identification in Multiple Object Tracking (2020, methodology): “Similarly, JDE (Wang et al. 2020b) is built on top of YOLOv3 (Redmon and Farhadi 2018) which achieves near video rate inference.”
BoT-SORT: Robust Associations Multi-Pedestrian Tracking (2022, methodology): “We adopt the exponential moving average (EMA) mechanism for updating the matched tracklets appearance state e ki for the i -th tracklet at frame k , as in [46], Eq.”
Crossover Learning for Fast Online Video Instance Segmentation (2021, methodology): “We then employ a set of learnable instance-wise weights { w M } := { w 1 , w 2 , . . . , w M } as proxies of instances (see Fig 3, right) to replace the embeddings of instances { e N } deﬁned by the sampled frame pair directly [44, 11, 65].”
AlphaPose: Whole-Body Regional Multi-Person Pose Estimation and Tracking in Real-Time (2022, methodology): “For the detectionmodule, we adopt YOLOX [81] YOLOV3-SPP [64], EfficientDet [75] and JDE [67].”
Quasi-Dense Similarity Learning for Multiple Object Tracking (2020, methodology): “Other data augmentation includes random horizontal flipping and color jittering, which is the common practice in [35, 44, 54].”
Joint Object Detection and Multi-Object Tracking with Graph Neural Networks (2021, methodology): “Following TR-MOT [27], we use a mix of six pedestrian detection datasets for pre-training, including CalTech [70], MOT17 [65], CUHK-SYSU [32], ETH [71], PRW [72], and CityPersons [73].”
TransTrack: Multiple-Object Tracking with Transformer (2020, methodology): “Most state-of-the-art multiobject trackers [43, 39, 7, 42, 47, 41, 4] follow the trackingby-detection paradigm.”
Track to Detect and Segment: An Online Multi-Object Tracker (2021, methodology): “The JDT methods, however, are confronted with two issues: (i) Although in most JDT works [29, 45, 25, 50] the backbone network is shared, detection is usually performed standalone without exploring tracking cues.”
MOTR: End-to-End Multiple-Object Tracking with TRansformer (2021, methodology): “In MOT16, we compare MOTR with JDE [39] and DeepSORT 2 [41], which achieve state-of-the-art performance among previous methods in MOTA and IDF1 metrics, respectively.”
Rethinking the Competition Between Detection and ReID in Multiobject Tracking (2020, methodology): “For a fair comparison, we use the same training datasets as the baseline tracker JDE [8], consisting of ETH [30], CityPerson [31], CalTech [32], MOT17 [33], CUDK-SYSU [34] and PRW [35].”
StrongSORT: Make DeepSORT Great Again (2022, methodology): “To solve this problem, we replace the feature bank mechanism with the feature updating strategy proposed in [52], which updates the appearance state e ti for the i -th tracklet at frame t in an exponential moving average (EMA) manner as follows: where f ti is the appearance embedding of the current…”