Learning Dynamic Memory Networks for Object Tracking

By Tianyu Yang, Antoni B. Chan. European Conference on Computer Vision, 2018

320 citations recorded by the source corpus.

Template-matching methods for visual tracking have gained popularity recently due to their comparable performance and fast speed. However, they lack effective ways to adapt to changes in the target object's appearance, making their tracking accuracy still far from state-of-the-art. In this paper, we propose a dynamic memory network to adapt the template to the target's appearance variations during tracking. An LSTM is used as a memory controller, where the input is the search feature map and the outputs are the control signals for the reading and writing process of the memory block. As the location of the target is at first unknown in the search feature map, an attention mechanism is applied to concentrate the LSTM input on the potential target. To prevent aggressive model adaptivity, we apply gated residual template learning to control the amount of retrieved memory that is used to combine with the initial template. Unlike tracking-by-detection methods where the object's information is maintained by the weight parameters of neural networks, which requires expensive online fine-tuning to be adaptable, our tracker runs completely feed-forward and adapts to the target's appearance changes by updating the external memory. Moreover, unlike other tracking methods where the model capacity is fixed after offline training --- the capacity of our tracker can be easily enlarged as the memory requirements of a task increase, which is favorable for memorizing long-term object information. Extensive experiments on OTB and VOT demonstrates that our tracker MemTrack performs favorably against state-of-the-art tracking methods while retaining real-time speed of 50 fps.

Subfields: Computer Vision and Pattern Recognition, Safety, Risk, Reliability and Quality

View Paper: https://www.semanticscholar.org/paper/26c50d272883fc8f72656526f915bb283f772b27

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Safety, Risk, Reliability and Quality

Citations in the corpus

GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild (2018, background): “Tracker [80]; meta-learning based tracker MetaTracker [61]; attentive tracker DAT [79]; and other convolutional neural networks based approaches MDNet [60], GOTURN [54],”
Regular Polytope Networks (2021, methodology): “At a high level, all these approaches can be framed as a nonparametric estimation method (like nearest neighbors) sitting on top of a high-powered parametric function (Faster R-CNN in the case of object detection [63], a face feature extractor in [60] and [61], and a SiamFC feature extractor [70] fo”
Learning Spatio-Temporal Transformer for Visual Tracking (2021, methodology): “Previous Siamese trackers [28, 59, 16, 7] only exploit the spatial information for tracking, while online methods [63, 66, 11, 3] use historical predictions for model updates.”
STMTrack: Template-free Visual Tracking with Space-time Memory Networks (2021, methodology): “Although MemTrack [58] uses a large amount of historical information in the tracking process, the memory reading operation controlled by a LSTM may lose useful information.”
Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking (2021, methodology): “In other tracking frameworks with update mechanisms [20, 39, 8, 58, 60, 3], previous prediction results are collected to incrementally update the tracking model.”
Ocean: Object-aware Anchor-free Tracking (2020, methodology): “MemDTC SiamMASK SiamRPN++ ATOM STN DiMP DiMP Ocean Ocean [46] [43] [21] [4] [37] [2] [2] offline online EAO ↑ 0.”
MovieChat: From Dense Token to Sparse Memory for Long Video Understanding (2023, background): “There are some prior works exploring memory models [59] in various vision tasks in videos, such as video object segmentation (VOS) [18,30,55,56], multi-object tracking (MOT) [2,14,29,73], visual object tracking (VOT) [38, 42, 77, 84], and action understanding [68].”
Deformable Siamese Attention Networks for Visual Object Tracking (2020, background): “Recent work attempted to enhance the target representation by integrating the features of previous targets [41, 14], but the discriminative context information from the background is ignored.”
Video Object Segmentation Using Space-Time Memory Networks (2019, background): “Recently, the memory networks have been applied to some vision problems such as personalized image captioning [25], visual tracking [41], movie understanding [23], and summarization[17].”
The Seventh Visual Object Tracking VOT2019 Challenge Results (2019, methodology): “This tracker extends MemTrack [101], which uses a dynamic memory network to maintain the appearance variations of the object as tracking proceeds, by introducing a Distractor Template Canceling (DTC) [102] scheme to cancel out wrong responses from the object template.”
Know Your Surroundings: Exploiting Scene Information for Object Tracking (2020, methodology): “Yang et al [49,50] use an LSTM to update the target model to account for changes in target appearance through a sequence.”
Unsupervised Deep Tracking (2019, methodology): “Trackers SiamFC DCFNet CFNet UDT DSiam EAST HP SA-Siam SiamPRN RASNet SACF Siam-tri RT-MDNet MemTrack StructSiam UDT+ [1] [54] [49] [14] [17] [13] [15] [27] [55] [59] [12] [19] [58] [60]”
Learning the Model Update for Siamese Trackers (2019, methodology): “Some recent approaches [15, 45] proposed alternative updating strategies for Siamese trackers.”
Towards More Flexible and Accurate Object Tracking with Natural Language: Algorithms and Benchmark (2021, background): “Recently, some researchers begin to collect the tracking results which can be used to dynamically update the target template and attain better results [62, 67].”