FEELVOS: Fast End-To-End Embedding Learning for Video Object Segmentation

By Paul Voigtlaender, Yuning Chai, Florian Schroff, Hartwig Adam, Bastian Leibe, Liang-Chieh Chen. 2019

451 citations recorded by the source corpus.

Many of the recent successful methods for video object segmentation (VOS) are overly complicated, heavily rely on fine-tuning on the first frame, and/or are slow, and are hence of limited practical use. In this work, we propose FEELVOS as a simple and fast method which does not rely on fine-tuning. In order to segment a video, for each frame FEELVOS uses a semantic pixel-wise embedding together with a global and a local matching mechanism to transfer information from the first frame and from the previous frame of the video to the current frame. In contrast to previous work, our embedding is only used as an internal guidance of a convolutional network. Our novel dynamic segmentation head allows us to train the network, including the embedding, end-to-end for the multiple object segmentation task with a cross entropy loss. We achieve a new state of the art in video object segmentation without fine-tuning with a J&F measure of 71.5% on the DAVIS 2017 validation set. We make our code and models available at https://github.com/tensorflow/models/tree/master/research/feelvos.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Occluded Video Instance Segmentation: A Benchmark (2021, methodology): “C V ] 2 F eb 2 02 To dissect the OVIS dataset, we conduct a thorough evaluation of 5 state-of-the-art algorithms whose code is publicly available, including FEELVOS [38], IoUTracker+ [49], MaskTrack R-CNN [49], SipMask [4], and STEm-Seg [1].”
Video Instance Segmentation (2019, methodology): “We evaluate two recent video object segmentation algorithms OSMN [35] and FEELVOS [31].”
Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation (2019, background): “The task of video object segmentation requires segmenting foreground objects in a class-agnostic fashion [43, 21, 36, 38], often by leveraging ground truth masks available for the first frame during inference [6, 32, 10, 19, 37].”
DMM-Net: Differentiable Mask-Matching Network for Video Object Segmentation (2019, unknown): “7 FEELVOS [38]∗ 65.”
Delving into the Cyclic Mechanism in Semi-supervised Video Object Segmentation (2020, unknown): “A few recent methods FEELVOS [7] and AGSS-VOS [3] mainly exploited the guidance from the initial reference and the last previous frame to enhance the segmentation accuracy with deliberately designed feature matching scheme or attention mechanism.”
MAST: A Memory-Augmented Self-Supervised Tracker (2020, unknown): “To leverage the temporal consistency between two adjacent frames, many propagation-based methods often try to establish dense correspondences with optical flow or metric learning [20, 21, 29, 41, 56].”
SipMask: Spatial Information Preservation for Fast Image and Video Instance Segmentation (2020, unknown): “FEELVOS [41] mask propagation 26.”
A Survey on Deep Learning Technique for Video Segmentation (2021, methodology): “38 GCNet [161] 71.4 69.3 73.5 y25 Dyenet [134] 69.1 67.3 71.0 2.4 KMN [162] 76.0 74.2 77.8 8.3 MVOS [138] 59.2 56.3 62.1 1.5 CFBI [163] 81.9 79.3 84.5 2.2 FEELVOS [139] 71.5 69.1 74.0 2.2 LWL [164] 70.8 68.2 73.5 15.6 MHP-VOS [140] 75.3 71.8 78.8 y0.”
URVOS: Unified Referring Video Object Segmentation Network with a Large-Scale Benchmark (2020, methodology): “Oﬄine methods alleviate this issue and reduce runtime [35, 32, 29, 22].”
Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation (2021, methodology): “Faster approaches have been proposed such as a more efﬁcient online learning algorithm [6, 7, 8], MRF graph inference [51], temporal CNN [52], capsule routing [53], tracking [11, 13, 15, 54, 55, 56, 57], embedding learning [10, 58, 59] and space-time matching [17, 18, 19, 20].”
SwiftNet: Real-time Video Object Segmentation (2021, methodology): “For the second aspect, efficient pixel-wise matching [13, 29, 34], region-wise distance measuring [26, 9, 4], and correlation filtering [3, 32] are deployed to reduce computations.”
Video Instance Segmentation 2019: A Winning Approach for Combined Detection, Segmentation, Classification and Tracking. (2019, methodology): “Current best performing VOS methods either propagate labels from the ﬁrst frame [23, 21], or detect and segment potential objects and then link these over time [13].”
Spatial Feature Calibration and Temporal Fusion for Effective One-stage Video Instance Segmentation (2021, unknown): “From Table 6, we can see that among existing fast video object segmentation (VOS) methods, OSMN [44] with ‘track-by-detect’ and FEELVOS [37] achieve relatively lower mask AP scores.”
XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model (2022, methodology): “While some methods [48,59,53,23,26,6] also include the first reference frame for global matching, the context is still limited and it becomes harder to match as the video progresses.”
Siam R-CNN: Visual Tracking by Re-Detection (2019, methodology): “Time per frame (seconds) J & F Ours OSMN [75] Ours (fastest) Ours (Fine-tuned) SiamMask [66] PReMVOS [44] STM-VOS [52] DyeNet [39] FEELVOS [63] OSVOS-S [47] RGMP [71] CINM [3] VideoMatch [29] OnAVOS [64] FAVOS [12] OSVOS [8]”