TransTrack: Multiple-Object Tracking with Transformer

By Pei Sun, Yi Jiang, Rufeng Zhang, Enze Xie, Jinkun Cao, Xinting Hu, Tao Kong, Zehuan Yuan, Changhu Wang, Ping Luo. arXiv.org, 2020

798 citations recorded by the source corpus.

In this work, we propose TransTrack, a simple but efficient scheme to solve the multiple object tracking problems. TransTrack leverages the transformer architecture, which is an attention-based query-key mechanism. It applies object features from the previous frame as a query of the current frame and introduces a set of learned object queries to enable detecting new-coming objects. It builds up a novel joint-detection-and-tracking paradigm by accomplishing object detection and object association in a single shot, simplifying complicated multi-step settings in tracking-by-detection methods. On MOT17 and MOT20 benchmark, TransTrack achieves 74.5\% and 64.5\% MOTA, respectively, competitive to the state-of-the-art methods. We expect TransTrack to provide a novel perspective for multiple object tracking. The code is available at: \url{https://github.com/PeizeSun/TransTrack}.

Subfields: Computer Vision and Pattern Recognition, Safety, Risk, Reliability and Quality

View Paper: https://www.semanticscholar.org/paper/1020da59ab3db2b3051fb558559d7fdcd2c7e57b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Safety, Risk, Reliability and Quality

Citations in the corpus

Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions (2021, methodology): “For example, some works [5, 82, 71, 55, 23, 41] model the vision task as a dictionary lookup problem with learnable queries, and use the Transformer decoder as a task-specific head on top of the CNN backbone.”
MOTR: End-to-End Multiple-Object Tracking with TRansformer (2021, methodology): “8 / / 35226 177270 2982 TransTrack[35] 65.”
Transformer-Based Visual Segmentation: A Survey (2023, methodology): “TransTrack [186] uses the object query from the last frame as a new track query and outputs tracking boxes from the shared decoder.”
PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis (2023, background): “…detection (Liu et al., 2021; Wang et al., 2021; 2022; Ge et al., 2023; Carion et al., 2020), semantic segmentation (Zheng et al., 2021; Xie et al., 2021; Strudel et al., 2021) and so on (Sun et al., 2020; Li et al., 2022b; Zhao et al., 2021; Liu et al., 2022; He et al., 2022; Li et al., 2022a).”
Visual Saliency Transformer (2021, background): “A bunch of works combined CNNs and transformers into hybrid architectures for object detection [3, 89], panoptic segmentation [62], lane shape prediction [40], multiple object tracking [59], and so on.”
TransReID: Transformer-based Object Re-Identification (2021, methodology): “The Transformer models are then extended to some popular computer-vision tasks including image processing [2], object detection [1] , semantic segmentation [46, 41], object tracking [30], etc.”
Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Tracking (2022, methodology): “Recently, Transformers [63] have been used in MOT [41, 60, 77] to learn deep representations with both visual information and trajectory encoded.”
Learning Spatio-Temporal Transformer for Visual Tracking (2021, methodology): “Recently, transformer has shown their great potential in vision tasks like image classiﬁcation [12], object detection [5], semantic segmentation [49], multiple object tracking [44, 35], etc.”
BinsFormer: Revisiting Adaptive Bins for Monocular Depth Estimation (2022, background): “Following it, various types of queries are proposed for different tasks, such as pose query [43] and track query [44], [45].”
TransFG: A Transformer Architecture for Fine-grained Recognition (2021, methodology): “Later, transformer models are further extended to other popular computer vision tasks such as object detection [3], segmentation [46, 4, 51], object tracking [36].”
A Survey on Vision Transformer (2020, background): “proposed TransTrack [207], which is an online joint-detection-and-tracking pipeline.”
SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers (2021, methodology): “Other works have also used Transformers in a variety of tasks such as tracking [61, 62], super-resolution [63], ReID [64], Colorization [65], Retrieval [66] and multi-modal learning [67, 68].”
SwinFusion: Cross-domain Long-range Learning for General Image Fusion via Swin Transformer (2022, background): “There are many Transformer-based models that have achieved impressive performance in various vision tasks, such as visual recognition [29], [63], [64], object detection [30], [65], [66], tracking [67]–[69], segmentation [31], [70], and image restoration [32], [33], [71].”
MAXIM: Multi-Axis MLP for Image Processing (2022, background): “Despite successful applications to many high-level tasks [4, 23, 56, 83, 87, 100], the efficacy of these global models on low-level enhancement and restoration problems has not been studied extensively.”
Deep OC-Sort: Multi-Pedestrian Tracking by Adaptive Re-Identification (2023, background): “More recently, the rise of transformers [14] has started another wave of using appearance for multi-object tracking, where the task of object association is modeled as a query matching problem [15, 16, 17, 13].”
DanceTrack: Multi-Object Tracking in Uniform Appearance and Diverse Motion (2021, methodology): “More recently, with the new focus of applying transformers [30] in vision tasks, TransTrack [27], TrackFormer [21] and MOTR [37] make attempts to leverage the attention mechanism in tracking objects in videos.”
AiATrack: Attention in Attention for Transformer Visual Tracking (2022, methodology): “We believe that the proposed AiA module could also be beneficial in other related tasks where the Transformer architecture can be applied to perform feature aggregation and information propagation, such as video object segmentation [60,33,15,39], video object detection [23] and multi-object tracking”