Learning Spatio-Temporal Transformer for Visual Tracking

By Bin Yan, Houwen Peng, Jianlong Fu, Dong Wang, Huchuan Lu. IEEE International Conference on Computer Vision, 2021

1,143 citations recorded by the source corpus.

In this paper, we present a new tracking architecture with an encoder-decoder transformer as the key component. The encoder models the global spatio-temporal feature dependencies between target objects and search regions, while the decoder learns a query embedding to predict the spatial positions of the target objects. Our method casts object tracking as a direct bounding box prediction problem, without using any proposals or predefined anchors. With the encoder-decoder transformer, the prediction of objects just uses a simple fully-convolutional network, which estimates the corners of objects directly. The whole method is end-to-end, does not need any postprocessing steps such as cosine window and bounding box smoothing, thus largely simplifying existing tracking pipelines. The proposed tracker achieves state-of-the-art performance on multiple challenging short-term and long-term benchmarks, while running at real-time speed, being 6× faster than Siam R-CNN [54]. Code and models are open-sourced at https://github.com/researchmm/Stark.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/72af9b2e03d3668e09edd0ec413b0b20cbce8f9c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework (2022, methodology): “While previous Transformer fusion trackers [5,29,47] random initialize the weights of Transformer layers, our joint feature learning and relation modeling module can directly benefit from the pre-trained weights.”
AiATrack: Attention in Attention for Transformer Visual Tracking (2022, methodology): “The update procedure is more efficient than the previous practices [19,53,58], as we directly reuse the encoded features.”
DETRs with Hybrid Matching (2022, methodology): “STARK [66, 78] built a spatial-temporal tracking scheme, based on DETR, to exploit spatial and temporal information simultaneously to address the single object tracking problem.”
Language as Queries for Referring Video Object Segmentation (2022, methodology): “Transformer Transformer [40] was first introduced for sequence-to-sequence translation in natural language processing community and has achieved marvelous success in most computer vision tasks [10, 17, 26] such as object detection [4, 55], tracking [5, 30, 37, 47] and segmentation [6, 15, 54].”
SwinTrack: A Simple and Strong Baseline for Transformer Tracking (2021, methodology): “[7] and [44] are the most advanced trackers in the visual object tracking task.”
PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking (2023, methodology): “An alternative strategy here would be to generate templates exclusively from timesteps with high visibility confidence, as commonly done in object tracking [63].”
Backbone is All Your Need: A Simplified Architecture for Visual Object Tracking (2022, methodology): “1, which replaces the CNN backbone of STARK-S [43] with a transformer backbone, and then show details of our SimTrack in Sec.”
ODTrack: Online Dense Temporal Token Learning for Visual Tracking (2024, methodology): “Video Sequence Sampling Strategy Most existing trackers (Yan et al. 2021a; Cui et al. 2022; Ye et al. 2022) commonly sample image-pairs within a short-term interval, such as 50, 100, or 200 frame intervals.”
Visual Prompt Multi-Modal Tracking (2023, result): “Overall performance on LasHeR test set [25]. mfDiMP [53] SGT [26] DAFNet [10] OSTrack [49] FANet [62] MaCNet [50] CAT [24] JMMAC [55] CMPP [42] APFNet [44] ProTrack [48] ViPT based tracker ProTrack [48], only has an improvement of 0.4% relative to their baseline STARK RGBD [46].”
Perception Test: A Diagnostic Benchmark for Multimodal Video Models (2023, methodology): “Grounded-vQA: As no existing model can perform this task, we created a baseline by running MDETR [34] on the middle frame and then tracking the predictions using the Stark [52] object tracker.”