MAST: A Memory-Augmented Self-Supervised Tracker

By Zihang Lai, Erika Lu, Weidi Xie. 2020

204 citations recorded by the source corpus.

Recent interest in self-supervised dense tracking has yielded rapid progress, but performance still remains far from supervised methods. We propose a dense tracking model trained on videos without any annotations that surpasses previous self-supervised methods on existing benchmarks by a significant margin (+15%), and achieves performance comparable to supervised methods. In this paper, we first reassess the traditional choices used for self-supervised training and reconstruction loss by conducting thorough experiments that finally elucidate the optimal choices. Second, we further improve on existing methods by augmenting our architecture with a crucial memory component. Third, we benchmark on large-scale semi-supervised video object segmentation~(aka. dense tracking), and propose a new metric: generalizability. Our first two contributions yield a self-supervised network that for the first time is competitive with supervised methods on standard evaluation metrics of dense tracking. When measuring generalizability, we show self-supervised approaches are actually superior to the majority of supervised methods. We believe this new generalizability metric can better capture the real-world use-cases for dense tracking, and will spur new interest in this research direction.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation (2021, background): “MAST [61] is an adjacent research that focused on unsupervised learning with a photometric reconstruction loss.”
A Survey on Deep Learning Technique for Video Segmentation (2021, methodology): “01 AFB-URR [154] 74.6 73.0 76.1 3.8 FAVOS [126] 58.2 54.6 61.8 0.56 Fasttan [155] 75.9 72.3 79.4 y7 MAST [160] 65.5 63.3 67.6 5.1 STM-Cycle [166] 71.7 68.7 74.7 38 CRW [174] 67.6 64.5 70.6 7.3 QMA [167] 71.9 - - 6.3 RGMP [127] 66.7 64.8 68.6 7.7 Fasttmu [156] 70.6 69.1 72.1 9.7 OSMN [128] 54.8 52.5 ”
CoTracker3: Simpler and Better Point Tracking by Pseudo-Labeling Real Videos (2025, unknown): “More robust unsupervised learning signals for long-range tracking can be obtained via simple colorization of grayscale videos by copying colors from the reference frame [38] or other visual patterns [17].”
Space-Time Correspondence as a Contrastive Random Walk (2020, methodology): “We also consider methods based on the Colorization approach of Vondrick et al. [83], including high-resolution methods: CorrFlow [43] and MAST [42].”
TAP-Vid: A Benchmark for Tracking Any Point in a Video (2022, unknown): “Although JHMDB is designed for human pose tracking [30], numerous prior works [27, 34, 35, 38, 80, 83, 86] use it as an evaluation for class-agnostic point tracking.”