Evaluating Multiple Object Tracking Performance: The CLEAR MOT Metrics

By Keni Bernardin, Rainer Stiefelhagen. 2008

3,170 citations recorded by the source corpus.

Simultaneous tracking of multiple persons in real-world environments is an active research field and several approaches have been proposed, based on a variety of features and algorithms. Recently, there has been a growing interest in organizing systematic evaluations to compare the various techniques. Unfortunately, the lack of common metrics for measuring the performance of multiple object trackers still makes it hard to compare their results. In this work, we introduce two intuitive and general metrics to allow for objective comparison of tracker characteristics, focusing on their precision in estimating object locations, their accuracy in recognizing object configurations and their ability to consistently label objects over time. These metrics have been extensively used in two large-scale international evaluations, the 2006 and 2007 CLEAR evaluations, to measure and compare the performance of multiple object trackers for a wide variety of tracking tasks. Selected performance results are presented and the advantages and drawbacks of the presented metrics are discussed based on the experience gained during the evaluations.

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, methodology): “We use standardized object tracking (MOT) metrics [18, 19] to evaluate speaker localization and tracking in the visual FoV.”
VirtualWorlds as Proxy for Multi-object Tracking Analysis (2016, methodology): “Our protocol rests on the comparison of real-world seed sequences with their corresponding synthetic clones using real-world pre-trained deep models (in particular Fast-RCNN [2]), hyper-parameter calibration via Bayesian optimization [3], and the analysis of taskspecific performance metrics [4].”
Performance Measures and a Data Set for Multi-target, Multi-camera Tracking (2016, methodology): “Single-camera, multi-object tracking performance is typically measured by the Multiple Object Tracking Accuracy (MOTA): and related scores (MOTP, MT, ML, FRG) [3–5].”
Scalable Object-Oriented Sequential Generative Models (2019, methodology): “Furthermore, we also measure standard object tracking metrics such as Multi Object Tracking Accuracy (MOTA), Multi Object Tracking Precision (MOTP) as well as Precision and Recall of the inferred bounding boxes (Bernardin & Stiefelhagen, 2008).”
StrongSORT: Make DeepSORT Great Again (2022, methodology): “We use the metrics MOTA, IDs, IDF1, HOTA, AssA, DetA and FPS to evaluate tracking performance [2, 40, 29].”
Chained-Tracker: Chaining Paired Attentive Regression Results for End-to-End Joint Multiple-Object Detection and Tracking (2020, methodology): “In the MOTChallenge benchmark, tracking performance is measured by the widely used CLEAR MOT Metrics [27], including Multiple-Object Tracking Accuracy (MOTA), Multiple-Object Tracking Precision (MOTP), the total number of False Negatives (FN), False Positives (FP), Identity Switches (IDS), and the…”
A Thermal Infrared Video Benchmark for Visual Analysis (2014, methodology): “To evaluate the tracking performance, the same “CLEAR MOT” metrics [15] were used.”
BoT-SORT: Robust Associations Multi-Pedestrian Tracking (2022, methodology): “Evaluations were performed according to the widely accepted CLEAR metrics [2].”
Multi-animal pose estimation, identification and tracking with DeepLabCut (2022, methodology): “MOTA is thereby the sum of three errors: the ratio of misses in the sequence, computed over the total number of objects present in all frames, the ratio of false positives and the ratio of mismatches 52 .”
CityFlow: A City-Scale Benchmark for Multi-Target Multi-Camera Vehicle Tracking and Re-Identification (2019, methodology): “The CLEAR MOT [6] metrics are used in the MOTChallenge benchmark for evaluating multiple object tracking performance.”
Online Multiperson Tracking-by-Detection from a Single, Uncalibrated Camera (2011, methodology): “We use the CLEAR MOT metrics [4] to evaluate the tracking performance.”
Person Re-identification: Past, Present and Future (2016, methodology): “In the multiple object tracking (MOT) benchmark [139], multiple evaluation metrics are used, including multiple object tracking precision (MOTP) [140], mostly track (MT) targets (percentage of ground truth persons whose trajectories are covered by the tracking results for at least 80%), the total…”
Visual Tracking: an Experimental Survey (2013, neutral): “(8) The literature is divided into two types of measures for precision and recall: one based on the localization of objects as a whole such as the F -score or OTA, and one based on the pixels.”
Self-Supervised Moving Vehicle Tracking With Stereo Sound (2019, background): “), false positive (FP) and false negative (FN) as evaluation metrics [10, 25].”