Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking

By Ning Wang, Wen-gang Zhou, Jie Wang, Houqiang Li. 2021

739 citations recorded by the source corpus.

In video object tracking, there exist rich temporal contexts among successive frames, which have been largely overlooked in existing trackers. In this work, we bridge the individual video frames and explore the temporal contexts across them via a transformer architecture for robust object tracking. Different from classic usage of the transformer in natural language processing tasks, we separate its encoder and decoder into two parallel branches and carefully design them within the Siamese-like tracking pipelines. The transformer encoder promotes the target templates via attention-based feature reinforcement, which benefits the high-quality tracking model generation. The transformer decoder propagates the tracking cues from previous templates to the current frame, which facilitates the object searching process. Our transformer-assisted tracking framework is neat and trained in an end-to-end manner. With the proposed transformer, a simple Siamese matching approach is able to outperform the current top-performing trackers. By combining our transformer with the recent discriminative tracking pipeline, our method sets several new state-of-the-art records on prevalent tracking benchmarks.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

In Model Architectures

Citations in the corpus

EDTER: Edge Detection with Transformer (2022, methodology): “Recently, it is applied in conjunction with CNN in DETR [7] and the other variants [10, 27, 31, 58, 73].”
Focal Self-attention for Local-Global Interactions in Vision Transformers (2021, methodology): “Beyond these static image tasks, it has also been applied on various temporal understanding tasks, such as action recognition [41, 83, 11], object tracking [15, 62], scene ﬂow estimation [39].”
Focal Modulation Networks (2022, background): “Its promise has been demonstrated in various vision tasks including image classification [63, 69, 74, 46, 87, 65], object detection [3, 97, 93, 15], segmentation [67, 72, 13], and beyond [38, 91, 4, 9, 68, 36].”
MPViT: Multi-Path Vision Transformer for Dense Prediction (2021, methodology): “classification [17, 37, 50, 51, 57, 58, 65, 67], object detection [5, 12, 74], and semantic segmentation [54, 63] to temporal tasks such as video classification [1, 3, 18] and object tracking [7, 41, 56].”
Focal Attention for Long-Range Interactions in Vision Transformers (2021, methodology): “Beyond these static image tasks, it has also been applied on various temporal understanding tasks, such as action recognition [40, 78, 10], object tracking [13, 59], scene flow estimation [38].”
SeqTrack: Sequence to Sequence Learning for Visual Object Tracking (2023, methodology): “For example, SiamRPN [27] and its follow-up works [3, 7, 48, 55, 58] adopt classification heads for object localization and regression heads for scale estimation, as sketched in Fig.”
SwinTrack: A Simple and Strong Baseline for Transformer Tracking (2021, background): “[41] improves the Siamese matching and DiMP based tracking frameworks by Transformer enhanced template features and search features.”
Backbone is All Your Need: A Simplified Architecture for Visual Object Tracking (2022, background): “The compared trackers include SiamFC [2](ECCVW16), ATOM [11] (CVPR19), SiamRPN++ [42](CVPR19), DiMP [3](ICCV19), SiamFC++ [42] (AAAI20), Ocean [49](ECCV20), SiamBAN [9](CVPR20), SiamAtt [45] (CVPR 20), TransT [38](CVPR21), TrDiMP [39](CVPR21), KeepTrack [27](ICCV21), STARKS [43](ICCV21), AutoMatch [”
Learning Target Candidate Association to Keep Track of What Not to Track (2021, methodology): “7%, exceeding the performance of the recent trackers TransT [6] and TrDiMP [43] by 0.”
AiATrack: Attention in Attention for Transformer Visual Tracking (2022, methodology): “However, in recent years, it has been noted that this benchmark has become highly saturated [53,58,41].”
A Survey on Vision Transformer (2020, background): “Some researchers also explored to use transformer encoder-decoder architecture in template-based discriminative trackers, such as TMT [229], TrTr [282] and TransT [33].”