TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild

By Matthias Müller, Adel Bibi, Silvio Giancola, Salman Al-Subaihi, Bernard Ghanem. 2018

1,118 citations recorded by the source corpus.

Despite the numerous developments in object tracking, further improvement of current tracking algorithms is limited by small and mostly saturated datasets. As a matter of fact, data-hungry trackers based on deep-learning currently rely on object detection datasets due to the scarcity of dedicated large-scale tracking datasets. In this work, we present TrackingNet, the first large-scale dataset and benchmark for object tracking in the wild. We provide more than 30K videos with more than 14 million dense bounding box annotations. Our dataset covers a wide selection of object classes in broad and diverse context. By releasing such a large-scale dataset, we expect deep trackers to further improve and generalize. In addition, we introduce a new benchmark composed of 500 novel videos, modeled with a distribution similar to our training dataset. By sequestering the annotation of the test set and providing an online evaluation server, we provide a fair benchmark for future development of object trackers. Deep trackers fine-tuned on a fraction of our dataset improve their performance by up to 1.6% on OTB100 and up to 1.7% on TrackingNet Test. We provide an extensive benchmark on TrackingNet by evaluating more than 20 trackers. Our results suggest that object tracking in the wild is far from being solved.

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “[162] Matthias Müller, Adel Bibi, Silvio Giancola, Salman Alsubaihi, and Bernard Ghanem.”
LaSOT: A High-quality Large-scale Single Object Tracking Benchmark (2020, methodology): “TrackingNet [67] is a large-scale benchmark with 30K sequences.”
Backbone is All Your Need: A Simplified Architecture for Visual Object Tracking (2022, methodology): “The same as STARK-S, we train our models with training-splits of LaSOT [10], GOT-10K [19], COCO2017 [26], and TrackingNet [30] for experiments on all testing datasets except for GOT-10k_Test.”
Visual Prompt Multi-Modal Tracking (2023, methodology): “RGB-based tracking, a foundation task of visual object tracking, gains from large-scale benchmarks [9, 13, 27, 36, 39, 45] provided by the community, and many excellent works [2, 3, 5, 7, 22] have spurted out over the past decades.”
Transformer Tracking (2021, methodology): “TrackingNet [30] is a large-scale tracking dataset, which covers diverse object classes and scenes.”
Deep Learning for Visual Tracking: A Comprehensive Survey (2019, background): “ther datasets), recent visual tracking benchmark datasets including OTB2013 [185], VOT [40]–[46], ALOV [48], OTB2015 [186], TC128 [187], UAV123 [188], NUS-PRO [189], NfS [190], DTB [191], TrackingNet [192], OxUvA [193], BUAA-PRO [194], GOT10k [195], and LaSOT [196] are compared. 4) Finally, extensiv”
Towards More Flexible and Accurate Object Tracking with Natural Language: Algorithms and Benchmark (2021, background): “Datasets #Videos #Min #Mean #Max #Total #FR #Attributes Aim Absent Lang-A Lang-I SAV Adv DA OTB50 [57] 51 71 578 3,872 29K 30 fps 11 Eval OTB100 [58] 100 71 590 3,872 59K 30 fps 11 Eval TC-128 [37] 128 71 429 3,872 55K 30 fps 11 Eval VOT-2017 [28] 60 41 356 1,500 21K 30 fps Eval NUS-PRO [29] 365 146”
Siam R-CNN: Visual Tracking by Re-Detection (2019, result): “2 shows our results on the TrackingNet test set [49] (511 videos, 442 frames average length), compared to five SOTA approaches.”
CoTracker3: Simpler and Better Point Tracking by Pseudo-Labeling Real Videos (2025, positive): “Therefore, the availability of high-quality training data is crucial for obtaining solid tracking results [26].”
Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking (2021, methodology): “We compare our proposed TrSiam and TrDiMP trackers with the recent state-of-the-art trackers on seven tracking benchmarks including TrackingNet [38], GOT-10k [23], LaSOT [13], VOT2018 [26], Need for Speed [24], UAV123 [36], and OTB-2015 [56].”
Know Your Surroundings: Exploiting Scene Information for Object Tracking (2020, methodology): “We use the training splits of TrackingNet [36], LaSOT [13], and GOT-10k [23] datasets to train our network.”
Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework (2022, methodology): “The training splits of COCO [30], LaSOT [14], GOT-10k [20] (1k forbidden sequences from GOT-10k training set are removed following the convention [47]) and TrackingNet [35] are used for training.”
Probabilistic Regression for Visual Tracking (2020, methodology): “The training splits of the La-SOT [13], GOT10k [23], TrackingNet [33] and COCO [30] are used, running 50 epochs with 1000 iterations each.”
Unsupervised Deep Tracking (2019, methodology): “Differently, in TrackingNet [36], forward-backward tracking is used for data annotation and tracker evaluation.”
SeqTrack: Sequence to Sequence Learning for Visual Object Tracking (2023, methodology): “Our training data includes the training splits of COCO [31], LaSOT [16], GOT-10k [20], and Track-ingNet [37].”