Deep Network Flow for Multi-object Tracking

By Samuel Schulter, Paul Vernaza, Wongun Choi, Manmohan Chandraker. 2017

213 citations recorded by the source corpus.

Data association problems are an important component of many computer vision applications, with multi-object tracking being one of the most prominent examples. A typical approach to data association involves finding a graph matching or network flow that minimizes a sum of pairwise association costs, which are often either hand-crafted or learned as linear functions of fixed features. In this work, we demonstrate that it is possible to learn features for network-flow-based data association via backpropagation, by expressing the optimum of a smoothed network flow problem as a differentiable function of the pairwise association costs. We apply this approach to multi-object tracking with a network flow formulation. Our experiments demonstrate that we are able to successfully learn all cost functions for the association problem in an end-to-end fashion, which outperform hand-crafted costs in all settings. The integration and combination of various sources of inputs becomes easy and the cost functions can be learned entirely from data, alleviating tedious hand-designing of costs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects (2018, methodology): “Schulter et al., 2017 used an end-to-end supervised approach that detects objects and performs data association.”
DeepEMD: Few-Shot Image Classification With Differentiable Earth Mover’s Distance and Structured Classifiers (2020, unknown): “[56] solve the multi-object tracking problem with a network flow formulation.”
Partially Does It: Towards Scene-Level FG-SBIR with Partial Input (2022, background): “Application of Optimal Transport The ability to learn structural similarity without explicit alignment information makes optimal transport [70] in linear optimisation an important tool for several downstream tasks [35,38,45,61,84].”
Features for Multi-target Multi-camera Tracking and Re-identification (2018, background): “end prediction of trajectories [52] or by learning data association by back-propagating through a network-flow solution [60].”
Track to Detect and Segment: An Online Multi-Object Tracker (2021, methodology): “MOT was dominated by the tracking-by-detection (TBD) paradigm over the past years [58, 6, 66, 52, 33, 5, 38, 48, 54].”
Evaluating Bayesian Deep Learning Methods for Semantic Segmentation (2018, background): “With the increasing use of AI systems in real-life scenarios like autonomous driving [42, 31, 7, 40, 58] and medical diagnosis [17, 61, 56, 45], there are many cases in which additional knowledge about the model’s confidence, i.”
Differentiation of Blackbox Combinatorial Solvers (2019, background): “Examples include multi-object tracking [41], semantic segmentation [7], multi-person pose estimation [35, 44], stereo matching [23] and person re-identification [52].”
Learning a Neural Solver for Multiple Object Tracking (2019, methodology): “This cost encodes the likelihood of the edge being active [43, 42, 67].”
Multiple object tracking: A literature review (2014, background): “The formulation and modeling of the target association problem using deep neural networks [201, 202, 203, 204] need more research efforts, although the first attempt to employ sequential neural networks for online MOT has been made very recently.”
Deep Affinity Network for Multiple Object Tracking (2018, background): “pre-trained for classification tasks are popular [57], [58], [59], [60], [61].”
DeepEMD: Differentiable Earth Mover's Distance for Few-Shot Learning (2020, background): “[99] solve the multi-object tracking problem with a network flow formulation that learns features for network-flow-based data association.”
SG-Net: Spatial Granularity Network for One-Stage Video Instance Segmentation (2021, background): “Recent multi-object tracking mainly follows the tracking-by-detection paradigm [9, 18, 34, 42, 48], which first detects objects and then predicts the temporal association of all detections.”
MOTR: End-to-End Multiple-Object Tracking with TRansformer (2021, methodology): “Dominant MOT methods mainly followed the tracking-by-detection paradigm [3,12,22,24,39].”
3D Multi-Object Tracking: A Baseline and New Evaluation Metrics (2019, methodology): “batch and online methods. The batch methods attempt to ﬁnd the global optimal solution from the entire sequence. They often create a network ﬂow graph and can be solved by the min-cost ﬂow algorithms [31,32]. On the other hand, the online methods consider only the detection at current frame and are ”
Tracking Objects as Points (2020, background): “Most modern trackers [2, 7, 23, 33, 35, 41, 47, 50, 58] follow the tracking-by-detection paradigm.”