A Novel Performance Evaluation Methodology for Single-Target Trackers

By Matej Kristan, Jiri Matas, Ales Leonardis, Tomás Vojír, Roman P. Pflugfelder, Gustavo Javier Fernandez, Georg Nebehay, Fatih Murat Porikli, Luka Cehovin. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015

661 citations recorded by the source corpus.

This paper addresses the problem of single-target tracker performance evaluation. We consider the performance measures, the dataset and the evaluation system to be the most important components of tracker evaluation and propose requirements for each of them. The requirements are the basis of a new evaluation methodology that aims at a simple and easily interpretable tracker comparison. The ranking-based methodology addresses tracker equivalence in terms of statistical significance and practical differences. A fully-annotated dataset with per-frame annotations with several visual attributes is introduced. The diversity of its visual properties is maximized in a novel way by clustering a large number of videos according to their visual attributes. This makes it the most sophistically constructed and annotated dataset to date. A multi-platform evaluation system allowing easy integration of third-party trackers is presented as well. The proposed evaluation methodology was tested on the VOT2014 challenge on the new dataset and 38 trackers, making it the largest benchmark to date. Most of the tested trackers are indeed state-of-the-art since they outperform the standard baselines, resulting in a highly-challenging benchmark. An exhaustive analysis of the dataset from the perspective of tracking difficulty is carried out. To facilitate tracker comparison a new performance visualization technique is proposed.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/0c7c61e2d85081bc4c63556f41d7bc71fdf0f5ac

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

An Empirical Study of Detection-Based Video Instance Segmentation (2019, background): “Visual object tracking [14] aims to localize an object captured in a video through its sequence of frames, with object’s initial location given in the first frame.”
TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild (2018, methodology): “Current trackers perform well on established datasets such as OTB [4, 5] and VOT [6–11] benchmarks.”
The Visual Object Tracking VOT2016 Challenge Results (2016, methodology): “In VOT2015 [18], the AR-raw plots from [19,20] were adopted to show the absolute average performance.”
Learning the Model Update for Siamese Trackers (2019, methodology): “We evaluate results on standard tracking benchmarks: VOT2018/16 [23], LaSOT [13] and TrackingNet [31].”
Staple: Complementary Learners for Real-Time Tracking (2015, methodology): “From VOT14 [25] we only include the top performers: DSST [8], SAMF [29], KCF [18], DGT [6], PLT 14 and PLT 13.”
LaSOT: A High-Quality Benchmark for Large-Scale Single Object Tracking (2018, background): “Currently, the popular dense benchmarks contain OTB [52, 53], TC-128 [35], VOT [25], NUS-PRO [28], UAV [39], NfS [14] and GOT-10k [22].”
Multi-cue Correlation Filters for Robust Visual Tracking (2018, methodology): “Finally, we evaluate the performance of the proposed trackers on the VOT2016 [21, 22] benchmark, which consists of 60 challenging sequences and provides an evaluation toolkit which will re-initialize the tracker to the correct position to continue tracking when tracking failure occurs.”
Learning Dynamic Siamese Network for Visual Object Tracking (2017, background): “There are two major dominant visual tracking strategies [19].”
Siam R-CNN: Visual Tracking by Re-Detection (2019, methodology): “Like our Siam R-CNN, many recent methods approach VOT using Siamese architectures.”
Opening up Open World Tracking (2021, background): “The related tasks of visual object tracking (VOT) [48, 47, 46] and segmentation (VOS) [82, 81, 14, 15] focus on tracking generic targets whose classes are not known a-priori, but are pre-localized as bounding box [48] or a segmentation mask [82] in the first frame.”
Discriminative Correlation Filter Tracker with Channel and Spatial Reliability (2016, methodology): “The VOT methodology [27] resets a tracker upon failure to fully use the dataset.”