Learning a Neural Solver for Multiple Object Tracking

By Guillem Brasó, L. Leal-Taix'e. Computer Vision and Pattern Recognition, 2019

486 citations recorded by the source corpus.

Graphs offer a natural way to formulate Multiple Object Tracking (MOT) within the tracking-by-detection paradigm. However, they also introduce a major challenge for learning methods, as defining a model that can operate on such structured domain is not trivial. As a consequence, most learning-based work has been devoted to learning better features for MOT and then using these with well-established optimization frameworks. In this work, we exploit the classical network flow formulation of MOT to define a fully differentiable framework based on Message Passing Networks (MPNs). By operating directly on the graph domain, our method can reason globally over an entire set of detections and predict final solutions. Hence, we show that learning in MOT does not need to be restricted to feature extraction, but it can also be applied to the data association step. We show a significant improvement in both MOTA and IDF1 on three publicly available benchmarks. Our code is available at https://bit.ly/motsolv.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/67e63dc78c79330d7edb14761b0c4a599600e534

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

FairMOT: On the Fairness of Detection and Re-identification in Multiple Object Tracking (2020, methodology): “MPNTrack (Brasó and Leal-Taixé, 2020) proposes trainable graph neural networks to perform a global association of the entire set of detections and make MOT fully differentiable.”
A Framework for Deprecating Datasets: Standardizing Documentation, Identification, and Communication (2021, background): “Duke MTMC, MegaFace and Tiny Images have alsobeen deprecatedbut are still widely circulated, used, and citedin bothindustry and academia (e.g [17, 55, 56, 105]).”
Rethinking the Competition Between Detection and ReID in Multiobject Tracking (2020, methodology): “Moreover, some works [17], [18] introduce graph neural networks to further improve matching capacity.”
MOTChallenge: A Benchmark for Single-Camera Multiple Target Tracking (2020, methodology): “To this end, we identiﬁed Graph Neural Network-based methods (Brasó and Leal-Taixé 2020) as a promising direction for future research.”
Joint Object Detection and Multi-Object Tracking with Graph Neural Networks (2021, methodology): “When comparing with prior disjoint MOT methods with relation modeling for data association such as [34], our method also performs better across all benchmarks, suggesting that only modeling object relations in data association is not enough, and that it is useful to model object relations in object ”
ByteTrack: Multi-Object Tracking by Associating Every Detection Box (2021, background): “A large number of methods [3,9,12,14,28,74,91] focus on improving the tracking performance based on these given detection results.”
TrackFormer: Multi-Object Tracking with Transformers (2021, methodology): “1 Furthermore, every MOT17 [13] frame is jittered by 1% with respect to the original image size similar to the adjacent frame simulation.”
Learning Target Candidate Association to Keep Track of What Not to Track (2021, background): “The dominant paradigm in MOT is tracking-bydetection [5, 1, 49, 40, 52], where tracking is posed as the problem of associating object detections over time.”
Track to Detect and Segment: An Online Multi-Object Tracker (2021, methodology): “MOT was dominated by the tracking-by-detection (TBD) paradigm over the past years [58, 6, 66, 52, 33, 5, 38, 48, 54].”
Space-Time Correspondence as a Contrastive Random Walk (2020, background): “This can be addressed in a variety of ways, including dynamic programming, particle ﬁltering, various graph-based combinatorial optimization, and more recently, graph neural networks [96, 69, 6, 66, 12, 94, 38, 37, 40, 9].”
Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection (2022, background): “Owing to their evident appearance discrepancy, it is challenging to fuse visually appealing images and/or to support higher-level vision tasks such as segmentation [4,29], tracking [2,7], and detection [32], by making full use of the complementary information from the infrared and visible.”
Opening up Open World Tracking (2021, background): “More recent efforts are largely data-driven, learning strong appearance models [35, 37], learning to regress targets [7] and to associate detections using graph neural networks [10].”
Towards Real-Time Multi-Object Tracking (2019, background): “Some recent works attempt to model the association problem using graph networks [4,20], so that end-to-end association can be achieved.”