Video Object Segmentation with Episodic Graph Memory Networks

By Xinkai Lu, Wenguan Wang, Martin Danelljan, Tianfei Zhou, Jianbing Shen, Luc Van Gool. European Conference on Computer Vision, 2020

302 citations recorded by the source corpus.

How to make a segmentation model efficiently adapt to a specific video and to online target appearance variations are fundamentally crucial issues in the field of video object segmentation. In this work, a graph memory network is developed to address the novel idea of "learning to update the segmentation model". Specifically, we exploit an episodic memory network, organized as a fully connected graph, to store frames as nodes and capture cross-frame correlations by edges. Further, learnable controllers are embedded to ease memory reading and writing, as well as maintain a fixed memory scale. The structured, external memory design enables our model to comprehensively mine and quickly store new knowledge, even with limited visual information, and the differentiable memory controllers slowly learn an abstract method for storing useful representations in the memory and how to later use these representations for prediction, via gradient descent. In addition, the proposed graph memory network yields a neat yet principled framework, which can generalize well both one-shot and zero-shot video object segmentation tasks. Extensive experiments on four challenging benchmark datasets verify that our graph memory network is able to facilitate the adaptation of the segmentation network for case-by-case video object segmentation.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/868474a3443f61e86914bff4b1d180a05e478bfa

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Region Graph Embedding Network for Zero-Shot Learning (2020, background): “Speciﬁcally, the PRR branch is based on GCN [21, 31] for relation reasoning.”
ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation (2022, unknown): “Extensive works have been proposed [8, 26, 30], e.”
MovieChat: From Dense Token to Sparse Memory for Long Video Understanding (2023, background): “There are some prior works exploring memory models [59] in various vision tasks in videos, such as video object segmentation (VOS) [18,30,55,56], multi-object tracking (MOT) [2,14,29,73], visual object tracking (VOT) [38, 42, 77, 84], and action understanding [68].”
A Survey on Deep Learning Technique for Video Segmentation (2021, background): “Each of the annotated object tracklets will be matched with one of predicted tracklets according to TABLE 7 Quantitative Object-Level AVOS Results on DAVIS16 [17] val (Section 5.1.2) in Terms of Region Similarity J , Boundary Accuracy F and Time Stability T Method J F T FPS" mean" recall" decay# mea”
STMTrack: Template-free Visual Tracking with Space-time Memory Networks (2021, background): “Recently, memory networks have shown significant performance improvement in few shot learning [39, 30], video object segmentation [37, 30], etc.”
Associating Objects with Transformers for Video Object Segmentation (2021, methodology): “, EGMN [23] and KMN [34]) leverage a memory network to embed past-frame predictions into memory and apply a non-local attention mechanism on the memory to decode the segmentation of the current frame.”
XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model (2022, background): “Particularly, SpaceTime Memory (STM) [36] is popular and has been extended by many follow-up works [43,8,18,54,50,31,9,44,33].”