MOTR: End-to-End Multiple-Object Tracking with TRansformer

By Fangao Zeng, Bin Dong, Tiancai Wang, Cheng Chen, X. Zhang, Yichen Wei. European Conference on Computer Vision, 2021

826 citations recorded by the source corpus.

Temporal modeling of objects is a key challenge in multiple object tracking (MOT). Existing methods track by associating detections through motion-based and appearance-based similarity heuristics. The post-processing nature of association prevents end-to-end exploitation of temporal variations in video sequence. In this paper, we propose MOTR, which extends DETR and introduces track query to model the tracked instances in the entire video. Track query is transferred and updated frame-by-frame to perform iterative prediction over time. We propose tracklet-aware label assignment to train track queries and newborn object queries. We further propose temporal aggregation network and collective average loss to enhance temporal relation modeling. Experimental results on DanceTrack show that MOTR significantly outperforms state-of-the-art method, ByteTrack by 6.5% on HOTA metric. On MOT17, MOTR outperforms our concurrent works, TrackFormer and TransTrack, on association performance. MOTR can serve as a stronger baseline for future research on temporal modeling and Transformer-based trackers. Code is available at https://github.com/megvii-research/MOTR.

Subfields: Automotive Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/b6d83dfbd8b748006cb05882674e82054e5bcc32

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Automotive Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Transformer-Based Visual Segmentation: A Survey (2023, methodology): “In particular, MOTR proposes a new tracklet-awared label assignment to train track queries and a temporal aggregation module to fuse temporal features.”
DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving (2024, neutral): “End-to-end autonomous driving has gained signiﬁcant attention in recent years due to advancements in perception models (detection [5, 17, 24, 42], tracking [54–56], online mapping [ 27, 28, 32], etc .)”
Deep OC-Sort: Multi-Pedestrian Tracking by Adaptive Re-Identification (2023, background): “More recently, the rise of transformers [14] has started another wave of using appearance for multi-object tracking, where the task of object association is modeled as a query matching problem [15, 16, 17, 13].”
DanceTrack: Multi-Object Tracking in Uniform Appearance and Diverse Motion (2021, methodology): “More recently, with the new focus of applying transformers [30] in vision tasks, TransTrack [27], TrackFormer [21] and MOTR [37] make attempts to leverage the attention mechanism in tracking objects in videos.”
ODTrack: Online Dense Temporal Token Learning for Visual Tracking (2024, background): “Multi-object tracking algorithms(Meinhardt et al. 2022; Zeng et al. 2022) typically involve the recognition and association of individual objects in a video, making the study of trajectory information a common practice.”
StrongSORT: Make DeepSORT Great Again (2022, methodology): “We train the detector on the CrowdHuman dataset [46] and MOT17 half training set for ablation following [50, 63, 70, 73, 77].”
Motion Transformer with Global Intention Localization and Local Movement Refinement (2022, methodology): “Transformer [48] has been widely applied in natural language processing [11, 2] and computer vision [13, 50, 4, 49, 58].”
YOLOv10: Real-Time End-to-End Object Detection (2024, background): “It is widely adopted in various practical applications, including autonomous driving [3], robot navigation [12], and object tracking [72], etc .”
DETRs Beat YOLOs on Real-time Object Detection (2023, methodology): “Real-time object detection is an important area of research and has a wide range of applications, such as object tracking [40], video surveillance [26], and autonomous driving [2], etc. Existing real-time detectors generally adopt the CNN-based architecture, the most famous of which is the YOLO…”
Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition (2021, background): “There are also attempts to adapt ViT for other vision tasks (e.g., object detection, semantic segmentation, etc.) [30, 44, 12, 20, 57, 60, 14].”
Planning-oriented Autonomous Driving (2022, methodology): “Inspired by [56, 58], we take a similar query design.”
Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection (2023, methodology): “MOTR [54] into multi-view 3D object tracking and achieve a promising result.”