VideoMatch: Matching based Video Object Segmentation

By Yuan-Ting Hu, Jia-Bin Huang, Alexander Gerhard Schwing. 2018

296 citations recorded by the source corpus.

Video object segmentation is challenging yet important in a wide variety of applications for video analysis. Recent works formulate video object segmentation as a prediction task using deep nets to achieve appealing state-of-the-art performance. Due to the formulation as a prediction task, most of these methods require fine-tuning during test time, such that the deep nets memorize the appearance of the objects of interest in the given video. However, fine-tuning is time-consuming and computationally expensive, hence the algorithms are far from real time. To address this issue, we develop a novel matching based algorithm for video object segmentation. In contrast to memorization based classification techniques, the proposed approach learns to match extracted features to a provided template without memorizing the appearance of the objects. We validate the effectiveness and the robustness of the proposed method on the challenging DAVIS-16, DAVIS-17, Youtube-Objects and JumpCut datasets. Extensive results show that our method achieves comparable performance without fine-tuning and is much more favorable in terms of computational time.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Occluded Video Instance Segmentation: A Benchmark (2021, background): “According to whether to provide the mask for the first frame, VOS can be divided into semi-supervised and unsupervised scenarios.”
See More, Know More: Unsupervised Video Object Segmentation With Co-Attention Siamese Networks (2019, methodology): “First, since [22] was proposed based on image matching strategy, they used a Siamese network to propagate the first-frame annotation to the subsequent frames.”
Video Instance Segmentation Tracking With a Modified VAE Architecture (2020, methodology): “Another Another line of methods [11, 23, 53] formulates VOS as a pixel-wise matching problem.”
MAST: A Memory-Augmented Self-Supervised Tracker (2020, unknown): “To leverage the temporal consistency between two adjacent frames, many propagation-based methods often try to establish dense correspondences with optical ﬂow or metric learning [20, 21, 29, 41, 56].”
Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation (2021, methodology): “Faster approaches have been proposed such as a more efficient online learning algorithm [8, 6, 7], MRF graph inference [51], temporal CNN [52], capsule routing [53], tracking [11, 13, 15, 54, 55, 56, 57], embedding learning [10, 58, 59] and space-time matching [17, 18, 19, 20].”
A Generative Appearance Model for End-To-End Video Object Segmentation (2018, methodology): “Also more explicit matching mechanisms have been proposed, where the input features are matched with a set of features with known class membership [4, 13] using KNearest-Neighbour (KNN).”
A Survey on Deep Learning Technique for Video Segmentation (2021, methodology): “RCAL [131] CVPR FCN + RL Propagation-based MSRA10K [84] + PASCAL-S + SOD + ECSSD [118] + DAVIS16 [17] OSVOS-S [132] PAMI FCN Online Fine-tuning DAVIS16 [17]/DAVIS17 [81] Videomatch [133] ECCV Siamese FCN Matching-based DAVIS16 [17]/DAVIS17 [81] Dyenet [134] ECCV Re-ID Propagation-based DAVIS17 [81] ”
Siam R-CNN: Visual Tracking by Re-Detection (2019, methodology): “Time per frame (seconds) J & F Ours OSMN [75] Ours (fastest) Ours (Fine-tuned) SiamMask [66] PReMVOS [44] STM-VOS [52] DyeNet [39] FEELVOS [63] OSVOS-S [47] RGMP [71] CINM [3] VideoMatch [29] OnAVOS [64] FAVOS [12] OSVOS [8]”
SAIL-VOS: Semantic Amodal Instance Level Video Object Segmentation – A Synthetic Dataset and Baselines (2019, methodology): “We employ the video object segmentation baseline Video-Match [46] in this experiment.”
SG-One: Similarity Guidance Network for One-Shot Semantic Segmentation (2018, methodology): “video segmentation methods [38], [41], [52] are derivatives of the Siamese network [17].”
FEELVOS: Fast End-To-End Embedding Learning for Video Object Segmentation (2019, methodology): “Note that this is not explicitly enforced, since instead of using distances in the embedding space directly to produce a segmentation like in PML [6] or VideoMatch [17], we use them as a soft cue which can be refined by the dynamic segmentation head.”
Video Object Segmentation Using Space-Time Memory Networks (2019, methodology): “Offline learning methods attempted to bypass the online learning while retaining the accuracy [24, 40, 3, 12, 13, 32, 33].”
Video Object Segmentation with Episodic Graph Memory Networks (2020, background): “tion (O-VOS)[5,48] and zero-shot video object segmentation (Z-VOS)[52]. Provided with only rst-frame annotations, O-VOS is to identify and segment the labeled object instances in the rest of the video[7,15,17,27,33,69]; whereas Z-VOS targets at automatically inferring the primary objects without any”
DMM-Net: Differentiable Mask-Matching Network for Video Object Segmentation (2019, methodology): “VideoMatch [15] applies a CNN to extract features from the reference and the current frames, respectively.”
The 2019 DAVIS Challenge on VOS: Unsupervised Multi-Object Segmentation (2019, methodology): “…methods that pushed the boundaries of video object segmentation not only in terms of accuracy [14], [15], [16], [17], [18], [19], [20] but also in terms of computational efﬁciency, giving rise to fast methods that can even operate online [21], [22], [23], [24], [25], [26], [27], [28], [29].”
Fast Online Object Tracking and Segmentation: A Unifying Approach (2018, background): “Recently, there has been an increasing interest in the VOS community towards faster methods [40, 63, 10, 9, 24, 23].”
Associating Objects with Transformers for Video Object Segmentation (2021, methodology): “Following PML and VideoMatch, FEELVOS [53] and CFBI [67, 68] extend the pixel-level matching mechanism by additionally matching between the current frame and the previous frame.”