MOTChallenge 2015: Towards a Benchmark for Multi-Target Tracking

By L. Leal-Taix'e, Anton Milan, Ian David Reid, Stefan Roth, Konrad Schindler. 2015

870 citations recorded by the source corpus.

In the recent past, the computer vision community has developed centralized benchmarks for the performance evaluation of a variety of tasks, including generic object and pedestrian detection, 3D reconstruction, optical flow, single-object short-term tracking, and stereo estimation. Despite potential pitfalls of such benchmarks, they have proved to be extremely helpful to advance the state of the art in the respective area. Interestingly, there has been rather limited work on the standardization of quantitative benchmarks for multiple target tracking. One of the few exceptions is the well-known PETS dataset, targeted primarily at surveillance applications. Despite being widely used, it is often applied inconsistently, for example involving using different subsets of the available data, different ways of training the models, or differing evaluation scripts. This paper describes our work toward a novel multiple object tracking benchmark aimed to address such issues. We discuss the challenges of creating such a framework, collecting existing and new data, gathering state-of-the-art methods to be tested on the datasets, and finally creating a unified evaluation system. With MOTChallenge we aim to pave the way toward a unified evaluation framework for a more meaningful quantification of multi-target tracking.

Subfields: Computer Vision and Pattern Recognition, Instrumentation, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Instrumentation

In Evaluation & Benchmarks

Citations in the corpus

Generalized Intersection Over Union: A Metric and a Loss for Bounding Box Regression (2019, background): “Due to this appealing property, all performance measures used to evaluate for segmentation [2, 1, 25, 14], object detection [14, 4], and tracking [11, 10] rely on this metric.”
Deep Perm-Set Net: Learn to predict sets with unknown permutation and cardinality using deep neural networks (2018, methodology): “We use training sequences from MOTChallenge pedestrian detection and tracking benchmark, i.e. 2DMOT2015 Leal-Taixé et al. (2015) and MOT17Det Milan et al. (2016), to create our dataset.”
MOT16: A Benchmark for Multi-Object Tracking (2016, methodology): “used metrics introduced in that work: mostly tracked, mostly lost, and partially tracked pedestrians. These numbers give a very good intuition on the performance of the method. We refer the reader to [28] for more formal deﬁnitions. A key parameter in both families of metrics is the intersection-ove”
Deep Attributes Driven Multi-Camera Person Re-identification (2016, methodology): “For the second stage training, we choose the MOT challenge[43] dataset to fine-tune dCNN O with attributes triplet loss.”
Deep Affinity Network for Multiple Object Tracking (2018, methodology): “ frames. 4.2 Multiple Object Tracking 15 (MOT15) In order to more comprehensively benchmark our technique for pedestrian tracking, we also evaluate it on Multiple Object Tracking 15 (MOT15) challenge [35] which also deals with multiple pedestrian tracking similar to MOT17. However, due to its earlie”
IntPhys: A Framework and Benchmark for Visual Intuitive Physics Reasoning (2018, background): “This is a well-studied problem with its own literature [51, 44, 55, 9] and challenges [32, 34].”
Simple online and realtime tracking with a deep association metric (2017, neutral): “On the MOT challenge dataset [13], SORT with a state-of-the-art people detector [14] ranks on average higher than MHT on standard detections.”
Online Multi-Object Tracking with Dual Matching Attention Networks (2018, methodology): “We consider the metrics used by the MOT benchmarks [35,28] for evaluation, which includes Multiple Object Tracking Accuracy (MOTA) [4], Multiple Object Tracking Precision (MOTP) [4], ID F1 score [41] (IDF, the ratio of correct detections over the average number of ground-truth and computed detection”
UA-DETRAC: A new benchmark and protocol for multi-object detection and tracking (2015, methodology): “1 In this work, we use the explicit deﬁnition of MOT system, i.e. , MOT sys-tem = detection + tracking . a r X i v : 1511 .”
Rethinking the Competition Between Detection and ReID in Multiobject Tracking (2020, methodology): “While in the MOT task [11, 12], features for ReID network are required to have scale-aware capability, since the objects’ size may change drastically across frames.”
The Seventh Visual Object Tracking VOT2019 Challenge Results (2019, methodology): “Tracking research has been historically promoted by several initiatives like PETS [105], CAVIAR 2 , i-LIDS 3 , ETISEO 4 , CDC [31], CVBASE 5 , FERET [77], LTDT 6 , MOTC [55, 83] and Videonet 7 .”
STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videos (2020, methodology): “Voigtlaender et al. [85] extended the KITTI [28] and MOTChallenge [46,56] datasets with pixel-precise annotations, and proposed a baseline method that adapts Mask R-CNN [32] to associate object proposals across time.”
DETRAC: A New Benchmark and Protocol for Multi-Object Tracking (2015, methodology): “However, existing performance evaluations of MOT methods usually separate the target association step from the object detection step by using the same object detection results as input for analysis.”
Online Multi-Target Tracking Using Recurrent Neural Networks (2016, methodology): “We further test our approach on real-world data, using the MOTChallenge 2015 benchmark (Leal-Taix´e et al. 2015).”
Tracking the Untrackable: Learning to Track Multiple Cues with Long-Term Dependencies (2017, methodology): “Our motion and interaction models leverage two separate Long Short-Term Memory (LSTM) networks that track the motion and interactions of objects for longer tracked period (suited in the presence of long-term occlusions).”
MOTS: Multi-Object Tracking and Segmentation (2019, background): “Many MOT datasets focus on street scenarios, for example the KITTI tracking dataset [13], which features video from a vehicle-mounted camera; or the MOTChallenge datasets [26, 37] that show pedestrians from a variety of different viewpoints.”
Person Re-identification: Past, Present and Future (2016, background): “In the multiple object tracking (MOT) benchmark [139], multiple evaluation metrics”
TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild (2018, methodology): “The tasks in the ﬁeld can be clustered between multi-object tracking [5, 6] and single-object tracking [18, 16].”