HOTA: A Higher Order Metric for Evaluating Multi-object Tracking

By Jonathon Luiten, Aljosa Osep, Patrick Dendorfer, Philip H. S. Torr, Andreas Geiger, L. Leal-Taix'e, Bastian Leibe. International Journal of Computer Vision, 2020

1,353 citations recorded by the source corpus.

Multi-object tracking (MOT) has been notoriously difficult to evaluate. Previous metrics overemphasize the importance of either detection or association. To address this, we present a novel MOT evaluation metric, higher order tracking accuracy (HOTA), which explicitly balances the effect of performing accurate detection, association and localization into a single unified metric for comparing trackers. HOTA decomposes into a family of sub-metrics which are able to evaluate each of five basic error types separately, which enables clear analysis of tracking performance. We evaluate the effectiveness of HOTA on the MOTChallenge benchmark, and show that it is able to capture important aspects of MOT performance not previously taken into account by established metrics. Furthermore, we show HOTA scores better align with human visual evaluation of tracking performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/6cdbe49de9f897530c34cd63e9ffa81d7f09b42a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

Kubric: A scalable dataset generator (2022, methodology): “Our final metric combines both classification accuracy and detection accuracy, and is inspired by the Jaccard-style metrics from the object tracking literature [60].”
TAP-Vid: A Benchmark for Tracking Any Point in a Video (2022, neutral): “Then, as has been advocated for object box tracking [41], we compute a Jaccard-style metric which incorporates both position and occlusion.”
Perception Test: A Diagnostic Benchmark for Multimodal Video Models (2023, methodology): “The answers are object tracks defined throughout the video and we use HOTA [39] metrics to evaluate performance.”
ByteTrack: Multi-Object Tracking by Associating Every Detection Box (2021, background): “For example, it outperforms TMOH [20] by 6.9 points on MOTA, 9.0 points on IDF1, 7.5 points on HOTA and reduce the identity switches by three quarters.”
BoT-SORT: Robust Associations Multi-Pedestrian Tracking (2022, background): “Including Multiple-Object Tracking Accuracy (MOTA), False Positive (FP), False Negative (FN), ID Switch (IDSW), etc. , IDF1 [35] and Higher-Order Tracking Accuracy (HOTA) [27] to evaluate different aspects of the detection and tracking performance.”
Opening up Open World Tracking (2021, background): “HOTA metric [45] for closed-world tracking.”
StrongSORT: Make DeepSORT Great Again (2022, methodology): “We use the metrics MOTA, IDs, IDF1, HOTA, AssA, DetA and FPS to evaluate tracking performance [2, 40, 29].”
MOTR: End-to-End Multiple-Object Tracking with TRansformer (2021, neutral): “The common metrics include Higher Order Metric for Evaluating Multi-object Tracking [17] (HOTA, AssA, DetA), Multiple-Object Tracking Accuracy (MOTA), Identity Switches (IDS) and Identity F1 Score (IDF1).”
Deep OC-Sort: Multi-Pedestrian Tracking by Adaptive Re-Identification (2023, methodology): “We follow the HOTA protocol[26] for quantitative evaluation, which provides a more comprehensive measurement of the tracking quality.”
DanceTrack: Multi-Object Tracking in Uniform Appearance and Diverse Motion (2021, background): “For the detailed definitions of these metrics, we refer to [2, 20, 26].”
Humans in 4D: Reconstructing and Tracking Humans with Transformers (2023, background): “Canonical examples in this category include SlowFast [18] and MViT [16].”