Video Object Segmentation Using Space-Time Memory Networks

By Seoung Wug Oh, Joon-Young Lee, Ning Xu, Seon Joo Kim. 2019

868 citations recorded by the source corpus.

We propose a novel solution for semi-supervised video object segmentation. By the nature of the problem, available cues (e.g. video frame(s) with object masks) become richer with the intermediate predictions. However, the existing methods are unable to fully exploit this rich source of information. We resolve the issue by leveraging memory networks and learn to read relevant information from all available sources. In our framework, the past frames with object masks form an external memory, and the current frame as the query is segmented using the mask information in the memory. Specifically, the query and the memory are densely matched in the feature space, covering all the space-time pixel locations in a feed-forward fashion. Contrast to the previous approaches, the abundant use of the guidance information allows us to better handle the challenges such as appearance changes and occlussions. We validate our method on the latest benchmark sets and achieved the state-of-the-art performance (overall score of 79.4 on Youtube-VOS val set, J of 88.7 and 79.2 on DAVIS 2016/2017 val set respectively) while having a fast runtime (0.16 second/frame on DAVIS 2016 val set).

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

URVOS: Unified Referring Video Object Segmentation Network with a Large-Scale Benchmark (2020, methodology): “We average the self-attentive features over words and derive the ﬁnal cross-modal feature as Memory Attention Module To leverage information in the mask predictions at the frames processed earlier, we extend the idea introduced in [22] and design a memory attention module.”
Retrieval-Enhanced Machine Learning (2022, methodology): “Similar approaches have also been used for video object segmentation [43] and video summarization [38].”
MovieChat: From Dense Token to Sparse Memory for Long Video Understanding (2023, background): “There are some prior works exploring memory models [59] in various vision tasks in videos, such as video object segmentation (VOS) [18,30,55,56], multi-object tracking (MOT) [2,14,29,73], visual object tracking (VOT) [38, 42, 77, 84], and action understanding [68].”
Memory Enhanced Global-Local Aggregation for Video Object Detection (2020, methodology): “Like our method that tries to aggregate information beyond a small local range, [29, 20, 28, 33] share a similar spirit and shows superior results in different areas.”
Language as Queries for Referring Video Object Segmentation (2022, background): “STM [38] leverages a memory to store the past object features and utilize the attention matching mechanism on the memory to guide the prediction of current frame.”
Video Object Segmentation with Episodic Graph Memory Networks (2020, methodology): “The query encoder takes an RGB query frame as input, while, for the memory encoder, input is an RGB support frame concatenated with the one-channel softmax object mask and one-hot label map [31].”
Transformer-Based Visual Segmentation: A Survey (2023, methodology): “• Video Object Segmentation: Recent approaches for VOS mainly focus on designing better memory-based matching meth-ods [339].”
Siam R-CNN: Visual Tracking by Re-Detection (2019, methodology): “Time per frame (seconds) J & F Ours OSMN [75] Ours (fastest) Ours (Fine-tuned) SiamMask [66] PReMVOS [44] STM-VOS [52] DyeNet [39] FEELVOS [63] OSVOS-S [47] RGMP [71] CINM [3] VideoMatch [29] OnAVOS [64] FAVOS [12] OSVOS [8]”
STMTrack: Template-free Visual Tracking with Space-time Memory Networks (2021, methodology): “Recently, memory networks have shown significant performance improvement in few shot learning [39, 30], video object segmentation [37, 30], etc.”
Delving into the Cyclic Mechanism in Semi-supervised Video Object Segmentation (2020, methodology): “We first demonstrate the effectiveness of cyclic training and gradient correction in Table 3, where the baseline method [5] is implemented and retrained by ourselves.”
STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videos (2020, methodology): “State-of-theart approaches to this task [8,95,99,79,93,89,80,78,32,58] usually involve online”
MAST: A Memory-Augmented Self-Supervised Tracker (2020, background): “In dense visual tracking, the popular memory-augmented models treat key frames as memory [45], and use attention mechanisms to read from the memory.”
Associating Objects with Transformers for Video Object Segmentation (2021, methodology): “In AOT-L, the predicted frames are stored into long-term memory per δ frames, following the memory reading strategy [26].”
Regular Polytope Networks (2021, background): “Among them, we mention [60]–[62] for cumulative learning of face appearance models from video stream, [63]–[67] for object detection, [68] for video object segmentation, and [69] for visual object tracking.”