A Generative Appearance Model for End-To-End Video Object Segmentation

By Joakim Johnander, Martin Danelljan, Emil Brissman, Fahad Shahbaz Khan, Michael Felsberg. 2018

199 citations recorded by the source corpus.

One of the fundamental challenges in video object segmentation is to find an effective representation of the target and background appearance. The best performing approaches resort to extensive fine-tuning of a convolutional neural network for this purpose. Besides being prohibitively expensive, this strategy cannot be truly trained end-to-end since the online fine-tuning procedure is not integrated into the offline training of the network. To address these issues, we propose a network architecture that learns a powerful representation of the target and background appearance in a single forward pass. The introduced appearance module learns a probabilistic generative model of target and background feature distributions. Given a new image, it predicts the posterior class probabilities, providing a highly discriminative cue, which is processed in later network modules. Both the learning and prediction stages of our appearance module are fully differentiable, enabling true end-to-end training of the entire segmentation pipeline. Comprehensive experiments demonstrate the effectiveness of the proposed approach on three video object segmentation benchmarks. We close the gap to approaches based on online fine-tuning on DAVIS17, while operating at 15 FPS on a single GPU. Furthermore, our method outperforms all published approaches on the large-scale YouTube-VOS dataset.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Occluded Video Instance Segmentation: A Benchmark (2021, background): “According to whether to provide the mask for the first frame, VOS can be divided into semi-supervised and unsupervised scenarios.”
The Seventh Visual Object Tracking VOT2019 Challenge Results (2019, methodology): “First, segmentation masks have been generated semi-automatically based on the video object segmentation method [41].”
A Survey on Deep Learning Technique for Video Segmentation (2021, methodology): “Core Architecture Flow Technical Feature Training Dataset 2017 OSVOS [18] CVPR FCN Online Fine-tuning DAVIS16 [17] MaskTrack [117] CVPR FCN ✓ Propagation-based ECSSD [118] + MSRA10K [84] + PASCAL- S [88] + DAVIS16 [17] CTN [119] CVPR FCN ✓ Propagation-based PASCAL VOC 2012 [103] VPN [120] CVPR Bilat”
SwiftNet: Real-time Video Object Segmentation (2021, methodology): “To expedite time-consuming pixel-wise matching, RGMP [35] computes similarity responses with convolutions; AGAME [11] discriminates object from background with a probabilistic generative appearance model; RANet [34] adopts cross-correlation on ranked pixel-wise features to match query with reference”
Video Object Segmentation Using Space-Time Memory Networks (2019, methodology): “Results for other methods are directly copied from [37, 13, 32, 35].”
Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation (2021, methodology): “Most current methods either ﬁt a model using the initial segmentation [4, 5, 6, 7, 8, 9] or leverage temporal propagation [10, 11, 12, 13, 14, 15, 16], particularly with spatio-temporal matching [17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27].”
Video Object Segmentation with Episodic Graph Memory Networks (2020, methodology): “MSK OSMN RGMP OnAVOS RVOS OSVOS S2S AGAME PreMVOS DMMNet STM Method [33] [66] [59] [50] [48] [5] [64] [17] [26] [70] [31] Ours”
Personalize Segment Anything Model with One Shot (2023, methodology): “Notably, our fine-tuning approach can even outperform AGSS [33] and AGAME [25] by +4.”
MAST: A Memory-Augmented Self-Supervised Tracker (2020, unknown): “Notation : CINM [3], OSVOS [6], AGAME [28], VOSwL [31], FAVOS [8], mgPFF [33], CorrFlow [37], DyeNet [39], PReMVOS [41].”
XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model (2022, methodology): “While some methods [48,59,53,23,26,6] also include the first reference frame for global matching, the context is still limited and it becomes harder to match as the video progresses.”