Towards More Flexible and Accurate Object Tracking with Natural Language: Algorithms and Benchmark

By Xiao Wang, Xiujun Shu, Zhipeng Zhang, Bo Jiang, Yaowei Wang, Yonghong Tian, Feng Wu. Computer Vision and Pattern Recognition, 2021

297 citations recorded by the source corpus.

Tracking by natural language specification is a new rising research topic that aims at locating the target object in the video sequence based on its language description. Compared with traditional bounding box (BBox) based tracking, this setting guides object tracking with high-level semantic information, addresses the ambiguity of BBox, and links local and global search organically together. Those benefits may bring more flexible, robust and accurate tracking performance in practical scenarios. However, existing natural language initialized trackers are developed and compared on benchmark datasets proposed for tracking-by-BBox, which can’t reflect the true power of tracking-by-language. In this work, we propose a new benchmark specifically dedicated to the tracking-by-language, including a large scale dataset, strong and diverse baseline methods. Specifically, we collect 2k video sequences (contains a total of 1,244,340 frames, 663 words) and split 1300/700 for the train/testing respectively. We densely annotate one sentence in English and corresponding bounding boxes of the target object for each video. We also introduce two new challenges into TNL2K for the object tracking task, i.e., adversarial samples and modality switch. A strong baseline method based on an adaptive local-global-search scheme is proposed for future works to compare. We believe this benchmark will greatly boost related researches on natural language guided tracking.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/8b8ea4bb311f4a46cb654bc4d34cb07f49ed45ca

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Huchuan Lu
3. Ming-Hsuan Yang
4. Wanli Ouyang
5. Xiaogang Wang
6. Haibin Ling
7. Houwen Peng
8. Yi Yang

Citations in the corpus

• SwinTrack: A Simple and Strong Baseline for Transformer Tracking (2021, methodology): “We conduct extensive experiments on five large-scale benchmarks to verify the effectiveness of SwinTrack, including LaSOT [11], LaSOT ext [10], TrackingNet [33], GOT-10k [19] and TNL2k [42].”
• Backbone is All Your Need: A Simplified Architecture for Visual Object Tracking (2022, methodology): “We compare our SimTrack with state-of-the-art trackers on five datasets, including LaSOT [10], TNL2K [41], TrackingNet [30], UAV123 [29] and GOT-10k [19].”
• Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, methodology): “Similar tasks defined on videos are termed Spatio-Temporal Video Grounding (STVG) [257] or Tracking by Natural Language [258–260].”
• SeqTrack: Sequence to Sequence Learning for Visual Object Tracking (2023, methodology): “TNL2K, NFS and UAV123.”
• ODTrack: Online Dense Temporal Token Learning for Visual Tracking (2024, methodology): “L achieve the best performance on TNL2K and OTB100 benchmarks, demonstrating the effectiveness of the temporal token propagation attention mechanism.”
• Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework (2022, methodology): “NFS, UAV123 and TNL2K.”