Video Object Segmentation without Temporal Information

By Kevis-Kokitsi Maninis, Sergi Caelles, Yuhua Chen, Jordi Pont-Tuset, L. Leal-Taix'e, Daniel Cremers, Luc Van Gool. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017

358 citations recorded by the source corpus.

Video Object Segmentation, and video processing in general, has been historically dominated by methods that rely on the temporal consistency and redundancy in consecutive video frames. When the temporal smoothness is suddenly broken, such as when an object is occluded, or some frames are missing in a sequence, the result of these methods can deteriorate significantly. This paper explores the orthogonal approach of processing each frame independently, i.e., disregarding the temporal information. In particular, it tackles the task of semi-supervised video object segmentation: the separation of an object from the background in a video, given its mask in the first frame. We present Semantic One-Shot Video Object Segmentation (OSVOS$^\mathrm {S}$S), based on a fully-convolutional neural network architecture that is able to successively transfer generic semantic information, learned on ImageNet, to the task of foreground segmentation, and finally to learning the appearance of a single annotated object of the test sequence (hence one shot). We show that instance-level semantic information, when combined effectively, can dramatically improve the results of our previous method, OSVOS. We perform experiments on two recent single-object video segmentation databases, which show that OSVOS$^\mathrm {S}$S is both the fastest and most accurate method in the state of the art. Experiments on multi-object video segmentation show that OSVOS$^\mathrm {S}$S obtains competitive results.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/a2fcb9088e6dca3ea850de871c08b6fa084a190e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation (2021, methodology): “Most current methods either ﬁt a model using the initial segmentation [4, 5, 6, 7, 8, 9] or leverage temporal propagation [10, 11, 12, 13, 14, 15, 16], particularly with spatio-temporal matching [17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27].”
Video Object Segmentation with Episodic Graph Memory Networks (2020, methodology): “6 Recall" 66.1 67.5 72.3 73.2 73.8 83.4 75.4 77.7 Decay# 24.3 21.0 18.0 28.2 27.0 15.8 26.6 19.6 Times (s) 0.13 0.028 1.8 1.8 7.0 0.07 13 0.13 Method OSVOS-S RANet FEELVOS CINM PReMVO DMMNet STM [27] [56] [49] [3] [26] [70] [31] Ours J&F Mean " 68.0 65.7 71.5 70.6 77.8 70.7 81.8 82.8 J”
Fast Online Object Tracking and Segmentation: A Unifying Approach (2018, methodology): “[39, 45, 60]), similarly to what happens in most tracking approaches.”
An Empirical Study of Detection-Based Video Instance Segmentation (2019, background): “Many recent advances have been made in this challenging area ( e.g ., [1, 5, 17, 18, 25, 26]).”
Siam R-CNN: Visual Tracking by Re-Detection (2019, methodology): “Many methods perform fine-tuning on the template masks [8, 47, 64, 39, 3, 44], which leads to strong results but is slow.”
XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model (2022, background): “Online learning approaches either train or fine-tune their networks at test-time and are therefore typically slow in inference [3,49,32].”
Video Instance Segmentation Tracking With a Modified VAE Architecture (2020, background): “In semi-supervised mode [9, 60, 46, 12, 21], the initial masks for objects of interest are provided in the ﬁrst frame.”
DMM-Net: Differentiable Mask-Matching Network for Video Object Segmentation (2019, methodology): “In [6, 25], a pretrained CNN is first fine-tuned to predict both the segmentation mask and contour per frame and then a boundary snapping step is applied to combine both results.”
Fast Video Object Segmentation by Reference-Guided Mask Propagation (2018, methodology): “[27] extended this idea by incorporating additional information from an auxiliary instance segmentation network [26].”
Online Model Distillation for Efficient Video Inference (2018, methodology): “We believe the JITNet architecture could be employed as a means to significantly accelerate online VOS methods like OnAVOS [44] or more recent OSVOS-S [28] (uses MRCNN predictions every frame).”
Video Object Segmentation Using Space-Time Memory Networks (2019, methodology): “In [2, 21], an object-specific detector learned by fine-tuning the deep networks at the test time is used to segment out the target object.”
A Survey on Deep Learning Technique for Video Segmentation (2021, methodology): “RCAL [131] CVPR FCN + RL Propagation-based MSRA10K [84] + PASCAL-S + SOD + ECSSD [118] + DAVIS16 [17] OSVOS-S [132] PAMI FCN Online Fine-tuning DAVIS16 [17]/DAVIS17 [81] Videomatch [133] ECCV Siamese FCN Matching-based DAVIS16 [17]/DAVIS17 [81] Dyenet [134] ECCV Re-ID Propagation-based DAVIS17 [81] ”
FEELVOS: Fast End-To-End Embedding Learning for Video Object Segmentation (2019, background): “OnAVOS [35, 34] and OSVOS-S [27] extend OSVOS by an online adaptation mechanism, and by semantic information from an instance segmentation network, respectively.”
A Generative Appearance Model for End-To-End Video Object Segmentation (2018, methodology): “To tackle the aforementioned challenges, the standard strategy is to invoke extensive iterative optimization in the first frame [1, 2, 21, 30], given the initial image-mask pair.”
SwiftNet: Real-time Video Object Segmentation (2021, background): “Additionally, time-consuming online fine-tuning are exploited in [2, 18, 31] to improve segmentation accuracy, which however is impractical for realtime application.”