Siamese Instance Search for Tracking

By Ran Tao, Efstratios Gavves, Arnold M. W. Smeulders. Computer Vision and Pattern Recognition, 2016

665 citations recorded by the source corpus.

In this paper we present a tracker, which is radically different from state-of-the-art trackers: we apply no model updating, no occlusion detection, no combination of trackers, no geometric matching, and still deliver state-of-the-art tracking performance, as demonstrated on the popular online tracking benchmark (OTB) and six very challenging YouTube videos. The presented tracker simply matches the initial patch of the target in the first frame with candidates in a new frame and returns the most similar patch by a learned matching function. The strength of the matching function comes from being extensively trained generically, i.e., without any data of the target, using a Siamese deep neural network, which we design for tracking. Once learned, the matching function is used as is, without any adapting, to track previously unseen targets. It turns out that the learned matching function is so powerful that a simple tracker built upon it, coined Siamese INstance search Tracker, SINT, which only uses the original observation of the target from the first frame, suffices to reach state-of-the-art performance. Further, we show the proposed tracker even allows for target re-identification after the target was absent for a complete video shot.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/4f445f3e44f2f2ffb431cf1414c59ccba5a0b27d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Large Margin Object Tracking with Circulant Feature Maps (2017, methodology): “It is compared with 9 up-to-date CNNs based trackers including C-COT [8], DeepSRDCF[6], HCF[21], HDT[23], STCT[28], CNN-SVM[15], SINT[25], FCNT[27] and SiameseFC[3].”
Learning to Track at 100 FPS with Deep Regression Networks (2016, methodology): “A related area of research is patch matching [14,38], which was recently used for tracking in [33], running at 4 fps.”
Fast Online Object Tracking and Segmentation: A Unifying Approach (2018, background): “Recently, a radically different approach has been introduced [4, 21, 56].”
Unveiling the Power of Deep Tracking (2018, methodology): “The SINT method [10] learns a similarity measure on oﬄine videos, and localize the target using the initial labeled sample.”
Deep Metric Learning with Hierarchical Triplet Loss (2018, background): “…object classiﬁcation [30,34,32], key-point descriptor learning [12,24], face veriﬁcation [22,20], person re-identiﬁcation [30,23], object tracking [29] and etc. Recently, there is a number of widely-used loss functions developed for deep metric learning, such as contrastive loss [27,6], triplet…”
Deep Learning for Visual Tracking: A Comprehensive Survey (2019, methodology): “ICCV 2015 HCFT [59] DAF OTB, LaSOT ICCV 2015 DeepSRDCF [60] DAF, HOG OTB, VOT2018 ECCV 2016 CCOT [64] DAF OTB, VOT2018 ECCVW 2016 SiamFC [66] DAF OTB, LaSOT CVPR 2016 SINT [67] DAF OTB, LaSOT CVPR 2016 MDNet [68] DAF OTB, LaSOT CVPR 2016 HDT [69] DAF OTB ICCV 2017, TIP 2019 PTAV [77], [78] DAF, HOG ”
LaSOT: A High-Quality Benchmark for Large-Scale Single Object Tracking (2018, background): “…to provide extensive baselines, comprising deep trackers ( e.g ., MDNet [42], TRACA [5], CFNet [50], SiamFC [4], StructSiam [59], DSiam [16], SINT [49] and VITAL [48]), correlation ﬁl-ter trackers with hand-crafted features ( e.g ., ECO HC [7], DSST [8], CN [11], CSK [19], KCF [20], fDSST [9],…”
Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net (2018, methodology): “[27] used siamese matching networks to perform tracking.”
SiamRPN++: Evolution of Siamese Visual Tracking With Very Deep Networks (2018, methodology): “The Siamese network based tracking algorithms [42, 1] formulate visual tracking as a cross-correlation problem and learn a tracking similarity map from deep models with a Siamese network structure, one branch for learning the feature presentation of the target, and the other one for the search area.”
Fully-Convolutional Siamese Networks for Object Tracking (2016, methodology): “All of the competitive methods above that train on video sequences (MDNet [9], SINT [30], GOTURN [28]), use training data belonging to the same ALOV/OTB/VOT domain used by the benchmarks.”
End-to-End Representation Learning for Correlation Filter Based Tracking (2017, methodology): “Recent works have focused on learning deep embeddings that can be used as universal object descriptors [3, 12, 28, 17, 5].”
Lucid Data Dreaming for Object Tracking (2017, methodology): “Since then, convnets have been used as improved features [11] [32] [58], and eventually to drive the tracking itself [18] [3] [53] [35] [36].”
High Performance Visual Tracking with Siamese Region Proposal Network (2018, methodology): “In this experiment, we compare our method with several representive trackers, including PTAV [11], CREST[31], SRDCF [8], SINT [33], CSR-DCF [23], Siamese-FC [4], Staple [3], CFNet [35] and DSST [9].”
GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild (2018, methodology): “r [56] separates domain speciﬁc layers of the network from domain agnostic ones, and the domain speciﬁc part is re-trained on each testing video to adapt to new environment. The siamese tracker [38], [60] and its variants [61], [69], [80] learn a high dimensional metric space between the exemplar an”
SCALOR: Generative World Models with Scalable Object Representations (2019, methodology): “…community using the same “bounding box”-representation approach proposed in SCALOR (Kosiorek et al., 2017; Ning et al., 2017; Nam & Han, 2016; Tao et al., 2016), but such methods use provided labels to tackle the problem of object tracking and thus are usually fully or semi-supervised and…”
The Unmanned Aerial Vehicle Benchmark: Object Detection and Tracking (2018, background): “SINT [42] does not update its models during tracking, which results in a limited performance.”
CREST: Convolutional Residual Learning for Visual Tracking (2017, methodology): “We compare our CREST tracker with 29 trackers from the OTB-2013 benchmark [45] and other 21 state-of-the-art trackers including KCF [17], MEEM [48], TGPR [13], DSST [7], RPT [27], MUSTer [20], LCT [31], HCFT [30], FCNT [42], SRDCF [9], CNN-SVM [19], DeepSRDCF [8], DAT [35], Staple [2], SRDCFdecon [1”
Learning Adaptive Discriminative Correlation Filters via Temporal Consistency Preserving Spatial Feature Selection for Robust Visual Object Tracking (2018, methodology): “For robust comparison, SINT [34] proposed to consider the tracking task as a verification matching problem solved by a Siamese network.”