TrackFormer: Multi-Object Tracking with Transformers

By Tim Meinhardt, Alexander Kirillov, L. Leal-Taix'e, Christoph Feichtenhofer. Computer Vision and Pattern Recognition, 2021

1,096 citations recorded by the source corpus.

The challenging task of multi-object tracking (MOT) requires simultaneous reasoning about track initialization, identity, and spatio-temporal trajectories. We formulate this task as a frame-to-frame set prediction problem and introduce TrackFormer, an end-to-end trainable MOT approach based on an encoder-decoder Transformer architecture. Our model achieves data association between frames via attention by evolving a set of track predictions through a video sequence. The Transformer decoder initializes new tracks from static object queries and autoregressively follows existing tracks in space and time with the conceptually new and identity preserving track queries. Both query types benefit from self- and encoder-decoder attention on global frame-level features, thereby omitting any additional graph optimization or modeling of motion and/or appearance. TrackFormer introduces a new tracking-by-attention paradigm and while simple in its design is able to achieve state-of-the-art performance on the task of multi-object tracking (MOT17) and segmentation (MOTS20). The code is available at https://github.com/timmeinhardt/TrackFormer

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/0357156aef567fb5b709222894ddea1ce5d4e721

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Language as Queries for Referring Video Object Segmentation (2022, methodology): “Transformer Transformer [40] was first introduced for sequence-to-sequence translation in natural language processing community and has achieved marvelous success in most computer vision tasks [10, 17, 26] such as object detection [4, 55], tracking [5, 30, 37, 47] and segmentation [6, 15, 54].”
Transformer-Based Visual Segmentation: A Survey (2023, background): “TrackFormer [185] is the ﬁrst to treat multi-object tracking as a set prediction problem by performing joint detection and tracking-by-attention.”
Uncertainty-Guided Transformer Reasoning for Camouflaged Object Detection (2021, background): “Recent works have tried to apply transformers to various vision tasks, such as object detection [2, 5, 66], image recognition [8,47], multi-object tracking [39,48], semantic segmentation [64], and human pose and mesh reconstruction [30].”
Learning Spatio-Temporal Transformer for Visual Tracking (2021, methodology): “Different from STARK exploiting temporal information by introducing an extra dynamic template, TrackFormer [41] encodes temporal information by updating the query embedding.”
Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection (2023, methodology): “Since DETR [4] is proposed in 2D object detection, the object query has been applied in many downstream tasks [54, 35, 57, 56, 12] to model the temporal interaction.”
ODTrack: Online Dense Temporal Token Learning for Visual Tracking (2024, background): “Multi-object tracking algorithms(Meinhardt et al. 2022; Zeng et al. 2022) typically involve the recognition and association of individual objects in a video, making the study of trajectory information a common practice.”
BinsFormer: Revisiting Adaptive Bins for Monocular Depth Estimation (2022, background): “Following it, various types of queries are proposed for different tasks, such as pose query [43] and track query [44], [45].”
Improve Vision Transformers Training by Suppressing Over-smoothing (2021, background): “…have demonstrated promising performance on various challenging computer vision tasks, including image classiﬁcation (Dosovitskiy et al., 2020), object detection (Zhu et al., 2020; Carion et al., 2020), multi-object tracking (Meinhardt et al., 2021), and video understanding (Bertasius et al., 2021).”
SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers (2021, methodology): “Other works have also used Transformers in a variety of tasks such as tracking [61, 62], super-resolution [63], ReID [64], Colorization [65], Retrieval [66] and multi-modal learning [67, 68].”
MPViT: Multi-Path Vision Transformer for Dense Prediction (2021, methodology): “classification [17, 37, 50, 51, 57, 58, 65, 67], object detection [5, 12, 74], and semantic segmentation [54, 63] to temporal tasks such as video classification [1, 3, 18] and object tracking [7, 41, 56].”
Rethinking Semantic Segmentation: A Prototype View (2022, background): “Interestingly, the others [20, 100] follow the good practice of Transformer in other fields [11, 82, 113] and adopt a pixelquery strategy (Fig.”
MOTR: End-to-End Multiple-Object Tracking with TRansformer (2021, methodology): “Though TrackFormer [22] and TransTrack [35] are also Transformer-based MOT frameworks, the temporal information is not fully explored due to the two-frame training.”
Instances as Queries (2021, result): “Trackformer [35] and Transtrack [41] build a query based multiple object tracktor upon DETR and Deformable DETR, respectively, and attain comparable results to the non-query based methods.”
Efficient Training of Visual Transformers with Small Datasets (2021, background): “Visual Transformers (VTs) are progressively emerging architectures in computer vision as an alternative to standard Convolutional Neural Networks (CNNs), and they have already been applied to many tasks, such as image classification [17, 53, 61, 35, 58, 60, 33, 59], object detection [4, 66, 14], seg”
Conditional Object-Centric Learning from Video (2021, methodology): “, 2021), and supervised attention-based object-centric models (Fuchs et al., 2019; Carion et al., 2020; Kamath et al., 2021; Meinhardt et al., 2021) in Appendix A.”
Track to Detect and Segment: An Online Multi-Object Tracker (2021, methodology): “Alter-natively, transformer is exploited to match tracklets [36, 26].”