Learning by Tracking: Siamese CNN for Robust Target Association

By L. Leal-Taix'e, Cristian Canton-Ferrer, Konrad Schindler. 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2016

449 citations recorded by the source corpus.

This paper introduces a novel approach to the task of data association within the context of pedestrian tracking, by introducing a two-stage learning scheme to match pairs of detections. First, a Siamese convolutional neural network (CNN) is trained to learn descriptors encoding local spatio-temporal structures between the two input image patches, aggregating pixel values and optical flow information. Second, a set of contextual features derived from the position and size of the compared input patches are combined with the CNN output by means of a gradient boosting classifier to generate the final matching probability. This learning approach is validated by using a linear programming based multi-person tracker showing that even a simple and efficient tracker may outperform much more complex models when fed with our learned matching probabilities. Results on publicly available sequences show that our method meets state-of-the-art standards in multiple people tracking.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/498d79c61a4f998e0e50af670885a1d5c391f6a2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Quasi-Dense Similarity Learning for Multiple Object Tracking (2020, methodology): “Appearance similarity in MOT To exploit instance appearance similarity to strengthen tracking and re-identify vanished objects, some methods directly use an independent model [2,20,22,29,38,40,45,49] or add an extra embedding head to the detector for end-to-end training [26, 44, 50, 53].”
Deep learning in video multi-object tracking: A survey (2019, background): “[110] proposed a Siamese CNN which received two stacked images as an input, and output the probability of both images belonging to the same person.”
Tracking the Untrackable: Learning to Track Multiple Cues with Long-Term Dependencies (2017, unknown): “Recently, deep visual representation of objects have been used for modeling appearance [20, 33, 78].”
MOTChallenge: A Benchmark for Single-Camera Multiple Target Tracking (2020, methodology): “SiameseCNN [Leal-Taixe et al., 2016] trains a siamese convolutional neural network to learn spatiotemporal embeddings based on object appearance and estimated optical flow using contrastive loss [Hadsell et al.”
Opening up Open World Tracking (2021, background): “More recent efforts are largely data-driven, learning strong appearance models [35, 37], learning to regress targets [7] and to associate detections using graph neural networks [10].”
Deep Affinity Network for Multiple Object Tracking (2018, unknown): “7 SiameseCNN [86] offline 28.”
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations (2020, methodology): “In the computer vision (CV) literature, deep metric learning (DML) has been used to great effect for learning visual representations [24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34].”
SGPN: Similarity Group Proposal Network for 3D Point Cloud Instance Segmentation (2017, methodology): “Siamese CNNs [6, 43, 3] are used on tasks such as tracking [22] and one-shot learning [20] by measuring the similarity of two input images.”
SG-Net: Spatial Granularity Network for One-Stage Video Instance Segmentation (2021, background): “Others learn tracking representations by exploiting the appearance similarity [33, 42, 35] or re-identify vanished objects [17, 36, 37].”
Online Multi-Object Tracking with Dual Matching Attention Networks (2018, methodology): “atch processing. Typically, these methods model the MOT task as a global optimization problem in various forms such as network ow [58,51,14], and multi-cut [47,48,46]. In contrast, online MOT methods [53,10,27] do not rely on detections from future frames and may not perform well when target objects”
Survey on Deep Neural Networks in Speech and Vision Systems (2019, methodology): “In [118, 119], Siamese CNN is used to generate information between adjacent images patches, which is used in tracking objects.”
End-to-End Representation Learning for Correlation Filter Based Tracking (2017, background): “Recent works have focused on learning deep embeddings that can be used as universal object descriptors [3, 12, 28, 17, 5].”
Learning a Neural Solver for Multiple Object Tracking (2019, background): “The authors of [37] use a siamese network to directly learn the costs between a pair of detections, while a mixture of CNNs and recurrent neural networks (RNN) is used for the same purpose in [57].”
Multi-Similarity Loss With General Pair Weighting for Deep Metric Learning (2019, background): “[36], zero-shot learning [42, 1, 15], visual tracking [19, 31] and person re-identification [41, 9].”
Hard-Aware Deeply Cascaded Embedding (2016, background): “Deep metric embedding has attracted increasing attention for various tasks , such as visual product search [6, 18, 23, 27, 35], face recognition [21, 32, 41, 4], local image descriptor learning [9, 2, 12, 24], person/vehicle reidentification [22, 7, 43, 17], zero-shot image classification [19, 45, 5”
Spatial-Temporal Relation Networks for Multi-Object Tracking (2019, background): “According to what kind of information is used to establishing the association between objects in different frames, the existing methods can be categorized into online methods [2, 12, 19, 27, 33, 35, 38, 47, 50, 58, 59, 9], and offline methods [7, 14, 36, 39, 40, 44, 51, 52, 53, 62].”
Online Multi-object Tracking Using CNN-Based Single Object Tracker with Spatial-Temporal Attention Mechanism (2017, unknown): “1% 4533 41873 404 792 SiameseCNN[28] 29.”
Tracking Without Bells and Whistles (2019, background): “Both appearance and short-term motion in the form of optical flow can be used as input to a Siamese neural network to decide whether two boxes belong to the same track or not [35].”