Tracking the Untrackable: Learning to Track Multiple Cues with Long-Term Dependencies

By Amir Sadeghian, Alexandre Alahi, Silvio Savarese. IEEE International Conference on Computer Vision, 2017

551 citations recorded by the source corpus.

The majority of existing solutions to the Multi-Target Tracking (MTT) problem do not combine cues over a long period of time in a coherent fashion. In this paper, we present an online method that encodes long-term temporal dependencies across multiple cues. One key challenge of tracking methods is to accurately track occluded targets or those which share similar appearance properties with surrounding objects. To address this challenge, we present a structure of Recurrent Neural Networks (RNN) that jointly reasons on multiple cues over a temporal window. Our method allows to correct data association errors and recover observations from occluded states. We demonstrate the robustness of our data-driven approach by tracking multiple targets using their appearance, motion, and even interactions. Our method outperforms previous works on multiple publicly available datasets including the challenging MOT benchmark.

Subfields: Computer Vision and Pattern Recognition, Global and Planetary Change

View Paper: https://www.semanticscholar.org/paper/1454b646866052ef61a838737bd74ae191210cf0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Global and Planetary Change

Citations in the corpus

Online Multi-Object Tracking with Dual Matching Attention Networks (2018, background): “In [42], Sadeghian et al. propose to exploit the LSTM network to account for appearance modeling, which takes images in the tracklet step-by-step and predicts the similarity score.”
MOTChallenge: A Benchmark for Single-Camera Multiple Target Tracking (2020, methodology): “AMIR (Sadeghian et al. 2017) leverages recurrent neural networks in order to encode appearance, motion, pedestrian interactions and learns to combine these sources of information.”
Memory Based Online Learning of Deep Representations from Video Streams (2017, background): “Few exceptions can operate online and use deep features [41, 42, 43, 44] but they still assume continuous motion and do not update the appearance.”
Transformer Networks for Trajectory Forecasting (2020, background): “Social models and context: Enabled by the flexibility of the LSTM machinery, best performance has been recently achieved by modelling the social interaction [4], [5], [25] among people and the scene context [9], [7], [6], aided by tracking dynamics [26] and the spatio-temporal relations among neighb”
Peeking Into the Future: Predicting Future Person Activities and Locations in Videos (2019, background): “There are previous works that take into account multiple cues in videos for tracking [11, 25] and group activity recognition [5, 28, 27].”
Real-Time Multiple People Tracking with Deeply Learned Candidate Selection and Person Re-Identification (2018, background): “Multiple cues, including motion, shape and object appearances, are fused to mitigate this problem [3, 4].”
Tracking Objects as Points (2020, methodology): “ed bounding box to be within the frame. To handle this, we separately predict the visible and amodal bounding boxes [42]. Further details on this can be found in the supplement. We follow prior works [32, 39, 41, 52, 55] to pretrain on external data. We train our network on the CrowdHuman [34] datas”
Quasi-Dense Similarity Learning for Multiple Object Tracking (2020, methodology): “Appearance similarity in MOT To exploit instance appearance similarity to strengthen tracking and re-identify vanished objects, some methods directly use an independent model [2,20,22,29,38,40,45,49] or add an extra embedding head to the detector for end-to-end training [26, 44, 50, 53].”
Deep learning for cellular image analysis (2019, methodology): “Deep learning approaches to object linkage perform well on general-purpose data-sets, by tracking a single object 131–133 , attempting to learn a similarity score directly from data for use in a linear-programming framework 134 , or treating tracking as a reinforcement learning problem 135 .”
GSM: Graph Similarity Model for Multi-Object Tracking (2020, methodology): “Generally, the existing works can be categorized into online meth-ods [Sadeghian et al. , 2017; Zhu et al. , 2018; Xu et al. , 2019; Bergmann et al. , 2019] and ofﬂine methods [Li et al. , 2009; Maksai and Fua, 2019].”
Video Instance Segmentation (2019, methodology): “Methods [26, 32, 28] under this setting usually take the “trackingby-detection” strategy.”
Spatial-Temporal Relation Networks for Multi-Object Tracking (2019, methodology): “Besides, aggregation information across frames is also verified to be beneficial for measuring similarity [46, 26, 35].”
Deep learning in video multi-object tracking: A survey (2019, methodology): “Among the other works that later used LSTMs there is [123], that used a LSTM with a fully-connected (FC) layer to fuse features extracted by 3 other LSTMs (as already explained in section 3.”
SG-Net: Spatial Granularity Network for One-Stage Video Instance Segmentation (2021, background): “Others learn tracking representations by exploiting the appearance similarity [33, 42, 35] or re-identify vanished objects [17, 36, 37].”
STGAT: Modeling Spatial-Temporal Interactions for Human Trajectory Prediction (2019, background): “As a challenging task, the human trajectory prediction has attracted considerable attention in computer vision [42, 23, 28, 21, 24, 39, 20] and robotics [5, 19] ﬁelds over the past few years.”