Transformer Tracking

By Xin Chen, Bin Yan, Jiawen Zhu, Dong Wang, Xiaoyun Yang, Huchuan Lu. 2021

1,338 citations recorded by the source corpus.

Correlation acts as a critical role in the tracking field, especially in recent popular Siamese-based trackers. The correlation operation is a simple fusion manner to consider the similarity between the template and the search region. However, the correlation operation itself is a local linear matching process, leading to lose semantic information and fall into local optimum easily, which may be the bottleneck of designing high-accuracy tracking algorithms. Is there any better feature fusion method than correlation? To address this issue, inspired by Transformer, this work presents a novel attention-based feature fusion network, which effectively combines the template and search region features solely using attention. Specifically, the proposed method includes an ego-context augment module based on self-attention and a cross-feature augment module based on cross-attention. Finally, we present a Transformer tracking (named TransT) method based on the Siamese-like feature extraction backbone, the designed attention-based fusion mechanism, and the classification and regression head. Experiments show that our TransT achieves very promising results on six challenging datasets, especially on large-scale LaSOT, TrackingNet, and GOT-10k benchmarks. Our tracker runs at approximatively 50 fps on GPU. Code and models are available at https://github.com/chenxin-dlut/TransT.

Subfields: Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Focal Self-attention for Local-Global Interactions in Vision Transformers (2021, methodology): “Beyond these static image tasks, it has also been applied on various temporal understanding tasks, such as action recognition [41, 83, 11], object tracking [15, 62], scene ﬂow estimation [39].”
A Survey of Visual Transformers (2021, methodology): “Additionally, the visual Transformers have also obtained great performances for other CV tasks, such as detection [30], segmentation [31], [32], tracking [33], generation [34], and enhancement [35].”
LasHeR: A Large-Scale High-Diversity Benchmark for RGBT Tracking (2021, neutral): “…verify the diversity and challenge of our proposed LasHeR dataset, we present the results of some RGB trackers on LasHeR dataset (RGB videos), LaSOT [8] and TrackingNet [63] in Table VI, including TransT [66], SiamFC++ [67], PrDiMP-50 [68],PrDiMP-18 [68], DiMP-50 [45], DiMP-18 [45] and ATOM [64].”
CoTracker: It is Better to Track Together (2023, background): “We show that, by training the network by unrolling several sliding windows, CoTracker achieves excellent long-term tracking performance as well.”
Change Detection on Remote Sensing Images Using Dual-Branch Multilevel Intertemporal Network (2023, background): “…datasets, the transformer-based models demonstrate equivalent or even superior performance as the convolutional counter-parts, including image classification [22], semantic segmentation [22], [41], object detection [22], [42], image generation [43], video retrieval [44], and target tracking [45].”
Focal Modulation Networks (2022, background): “Its promise has been demonstrated in various vision tasks including image classification [63, 69, 74, 46, 87, 65], object detection [3, 97, 93, 15], segmentation [67, 72, 13], and beyond [38, 91, 4, 9, 68, 36].”
AiATrack: Attention in Attention for Transformer Visual Tracking (2022, background): “The crucial components in a typical Transformer tracking framework [53,8,61] are the attention blocks.”
ODTrack: Online Dense Temporal Token Learning for Visual Tracking (2024, methodology): “…are proposed, such as prediction head networks (Li et al. 2018; Chen et al. 2020; Zhang et al. 2020), cross-correlation modules (Han et al. 2021; Liao et al. 2020; Chen et al. 2021), powerful backbone (Chen et al. 2022; Cui et al. 2022) and attention mechanisms (Guo et al. 2021; Yu et al. 2020).”
SeqTrack: Sequence to Sequence Learning for Visual Object Tracking (2023, methodology): “Prior knowledge, such as window penalty [2,27] and online update [12, 28, 53], has been widely incorporated into existing tracking models and proved effective [7,10,43,55].”
Focal Attention for Long-Range Interactions in Vision Transformers (2021, methodology): “Beyond these static image tasks, it has also been applied on various temporal understanding tasks, such as action recognition [40, 78, 10], object tracking [13, 59], scene ﬂow estimation [38].”
A Survey on Vision Transformer (2020, background): “Some researchers also explored to use transformer encoder-decoder architecture in template-based discriminative trackers, such as TMT [229], TrTr [282] and TransT [33].”
DETRs with Hybrid Matching (2022, background): “com tion [7, 64, 59], pose estimation [54, 24, 53], multi-object tracking [6, 43, 55], monocular depth estimation [27, 17], text detection & layout analysis [49, 50, 40, 77], line segment detection [65], 3D object detection based on point clouds or multi-view images [46, 1, 61, 28], visual question ”