Learning Video Object Segmentation with Visual Memory

By Pavel Tokmakov, Alahari Karteek, Cordelia Schmid. 2017

342 citations recorded by the source corpus.

This paper addresses the task of segmenting moving objects in unconstrained videos. We introduce a novel two-stream neural network with an explicit memory module to achieve this. The two streams of the network encode spatial and temporal features in a video sequence respectively, while the memory module captures the evolution of objects over time. The module to build a “visual memory” in video, i.e., a joint representation of all the video frames, is realized with a convolutional recurrent unit learned from a small number of training video sequences. Given a video frame as input, our approach assigns each pixel an object or background label based on the learned spatio-temporal features as well as the "visual memory" specific to the video, acquired automatically without any manually-annotated frames. The visual memory is implemented with convolutional gated recurrent units, which allows to propagate spatial information over time. We evaluate our method extensively on two benchmarks, DAVIS and Freiburg-Berkeley motion segmentation datasets, and show state-of-the-art results. For example, our approach outperforms the top method on the DAVIS dataset by nearly 6%. We also provide an extensive ablative analysis to investigate the influence of each component in the proposed framework.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Glimpse Clouds: Human Activity Recognition from Unstructured Feature Points (2018, neutral): “In [53], visual memory is used to learn a spatio-temporal representation of moving objects in a scene.”
YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark (2018, methodology): “[13] use a two-stream network to model objects’ appearance and motion and use a recurrent layer to capture the evolution.”
Video Instance Segmentation (2019, background): “In unsupervised scenario, a single foreground object is segmented [29, 14, 30].”
Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation (2019, background): “The task of video object segmentation requires segmenting foreground objects in a class-agnostic fashion [43, 21, 36, 38], often by leveraging ground truth masks available for the first frame during inference [6, 32, 10, 19, 37].”
Robust Lane Detection From Continuous Driving Scenes Using Deep Neural Networks (2019, methodology): “In [59], a convolutional GRU was introduced to fuse the two-stream”
AMP: Adaptive Masked Proxies for Few-Shot Segmentation (2019, methodology): “Measure FSeg [10] LVO [36] MOTAdapt [30] ARP [13] PDB [33] AMP + CRF (Ours) J Mean 70.7 75.9 77.2 76.2 77.2 78.9 Recall 83.5 89.1 87.8 91.1 90.1 91.6 Decay 1.5 7.0 5.0 7.0 0.9 4.7 F Mean 65.3 72.1 77.4 70.6 74.5 78.4 Recall 73.8 83.4 84.4 83.5 84.4 87.3 Decay 1.8 1.3 3.3 7.9 0.2 2.7 Table 6: Quantit”
Video Object Detection with an Aligned Spatial-Temporal Memory (2017, methodology): “[41] used ConvGRUs for the task of video object segmentation.”
SAIL-VOS: Semantic Amodal Instance Level Video Object Segmentation – A Synthetic Dataset and Baselines (2019, background): “A variety of methods have tackled the unsupervised ( i.e ., no object annotation available during testing) [73, 71, 27, 44, 95, 96, 48] and the semi-supervised ( i.e ., ﬁrst frame annotation available during testing) [12, 54, 49, 11, 53, 98, 45, 18, 63, 105, 17, 15, 102, 100] setting.”
Pyramid Dilated Deeper ConvLSTM for Video Salient Object Detection (2018, background): “ARP*[20] LVO[34] FSEG[19] LMP[33] SFL*[5] FST*[30] SAGE*[42] Ours Ours+”
A Survey of Human-in-the-loop for Machine Learning (2021, background): “This research area has become popular in the computer vision community [151, 152, 153].”
Fast Video Object Segmentation by Reference-Guided Mask Propagation (2018, methodology): “We argue that our network structure, especially the refinement module used in the decoder, is able to recover fine details without additional post-processing, unlike previous methods [30, 21, 36] that are based on the DeepLab architecture [6] that produces the output at a coarser scale (1/8 compared”
Zero-Shot Video Object Segmentation via Attentive Graph Neural Networks (2019, methodology): “The correlation between two frames is efficiently captured by an attention mechanism, which avoids time-consuming optical flow estimation [7, 57, 21, 58, 31].”
Efficient Video Object Segmentation via Network Modulation (2018, methodology): “With the success of FCN on static image segmentation [12], deep learning based methods [23, 2, 32, 34, 18, 4] have been recently proposed for video segmentation and promising results have been achieved.”
SwiftNet: Real-time Video Object Segmentation (2021, methodology): “For the latter one, methods proposed in [20, 36, 28, 8, 27, 44, 23] leverage all previous frames and reveal improved accuracy, but they suffer slower speed for heavy computation overhead even with periodic sampling.”
Video Object Segmentation with Episodic Graph Memory Networks (2020, methodology): “LTV FST COSEG ARP LVO PDB FSEG SFL MotAdapt LSMO AGS COSNet AGNN Method [30] [32] [47] [19] [45] [42] [16] [8] [41] [46] [55] [24] [52] Ours”
Video Panoptic Segmentation (2020, methodology): “It combines several existing tasks: video object segmentation [1, 3, 4, 27, 30, 35, 36, 39] and video object detection [9,42,43], and aims at simultaneous detection, segmentation, and tracking of instances in videos.”
Learning Unsupervised Video Object Segmentation Through Visual Attention (2019, background): “Later, with the renaissance of neural network, many deep learning based models were proposed, which typically use multilayer perceptron based moving objectness detector, adopt two-stream architecture [67, 33], or CNN encoder-decoder structure [66, 11, 45, 46, 64].”
Instance Embedding Transfer to Unsupervised Video Object Segmentation (2018, methodology): “The advantage of combining motion and objectness is reported in several previous methods as well [6, 16, 37].”
URVOS: Unified Referring Video Object Segmentation Network with a Large-Scale Benchmark (2020, background): “They typically employ two-stream networks to fuse motion and appearance cues [27, 13, 38] for learning spatio-temporal representations.”