Video Object Segmentation with Joint Re-identification and Attention-Aware Mask Propagation

By Xiaoxiao Li, Chen Change Loy. 2018

206 citations recorded by the source corpus.

The problem of video object segmentation can become extremely challenging when multiple instances co-exist. While each instance may exhibit large scale and pose variations, the problem is compounded when instances occlude each other causing failures in tracking. In this study, we formulate a deep recurrent network that is capable of segmenting and tracking objects in video simultaneously by their temporal continuity, yet able to re-identify them when they re-appear after a prolonged occlusion. We combine both temporal propagation and re-identification functionalities into a single framework that can be trained end-to-end. In particular, we present a re-identification module with template expansion to retrieve missing objects despite their large appearance changes. In addition, we contribute a new attention-based recurrent mask propagation approach that is robust to distractors not belonging to the target segment. Our approach achieves a new state-of-the-art global mean (Region Jaccard and Boundary F measure) of 68.2 on the challenging DAVIS 2017 benchmark (test-dev set), outperforming the winning solution which achieves a global mean of 66.1 on the same partition.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

The 2019 DAVIS Challenge on VOS: Unsupervised Multi-Object Segmentation (2019, methodology): “This extended version triggered another wave of deep learning-based methods that pushed the boundaries of video object segmentation not only in terms of accuracy [14], [15], [16], [17], [18], [19], [20] but also in terms of computational efficiency, giving rise to fast methods that”
See More, Know More: Unsupervised Video Object Segmentation With Co-Attention Siamese Networks (2019, methodology): “In addition, both object tracking [29, 8, 12, 36] and person re-identification [34, 66] have been fused into SVOS task to handle deformation and occlusion issues.”
Dual Embedding Learning for Video Instance Segmentation (2019, background): “Most approaches [2, 11, 8] for video object segmentation task rely on fine-tuning the first-frame ground truth.”
Temporal Action Detection with Structured Segment Networks (2017, background): “2011), and video object segmentation (Li and Loy 2018), which are beyond the scope of this paper.”
A Generative Appearance Model for End-To-End Video Object Segmentation (2018, background): “proposed DyeNet [18], which combines optical flow with an object proposal network, interleaving bidirectional mask-propagation and target reidentification.”
MAST: A Memory-Augmented Self-Supervised Tracker (2020, unknown): “Notation : CINM [3], OSVOS [6], AGAME [28], VOSwL [31], FAVOS [8], mgPFF [33], CorrFlow [37], DyeNet [39], PReMVOS [41].”
Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation (2021, methodology): “Most current methods either ﬁt a model using the initial segmentation [4, 5, 6, 7, 8, 9] or leverage temporal propagation [10, 11, 12, 13, 14, 15, 16], particularly with spatio-temporal matching [17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27].”
Video Object Segmentation Using Space-Time Memory Networks (2019, methodology): “…s GAM (+YV) [15] 82.0 - 0.07 s FAVOS [4] 82.4 79.5 1.8 s LSE [7] 82.9 80.3 - CINN [1] 83.4 85.0 > 30 s PReMVOS [14] 84.9 88.6 > 30 s OSVOS S [22] 85.6 86.4 4.5 s OnAVOS [35] 86.1 84.9 13 s DyeNet [20] 86 Note that we use the network output directly without post-processing to evaluate our method.”
Siam R-CNN: Visual Tracking by Re-Detection (2019, methodology): “Many methods perform fine-tuning on the template masks [8, 47, 64, 39, 3, 44], which leads to strong results but is slow.”
Attention, please! A survey of neural attention models in deep learning (2021, background): “(2019g), Kong and Fowlkes (2019), Li and Loy (2018) and Zhao et al.”
URVOS: Unified Referring Video Object Segmentation Network with a Large-Scale Benchmark (2020, background): “Most approaches rely on online learning, which ﬁne-tunes networks using the ﬁrst-frame ground-truth at test-time [2, 14, 24].”
A Survey on Deep Learning Technique for Video Segmentation (2021, methodology): “20 18 CINM [125] CVPR Spatio-temporal MRF X Propagation-based DAVIS17 [81] FAVOS [126] CVPR FCN Propagation-based DAVIS16 [17]/DAVIS17 [81] RGMP [127] CVPR Siamese FCN Propagation-based PASCAL VOC 2012 [103] + ECSSD [118] + MSRA10K [84] + DAVIS17 [81] OSMN [128] CVPR FCN + Meta Learning Online Fine-”
XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model (2022, methodology): “While some methods [48,59,53,23,26,6] also include the first reference frame for global matching, the context is still limited and it becomes harder to match as the video progresses.”
Video Instance Segmentation Tracking With a Modified VAE Architecture (2020, methodology): “Some of the CNNbased methods [21, 41] employ Recurrent Neural Networks with optical flow to capture the temporal coherence of object motion and propagate information between frames.”