DMM-Net: Differentiable Mask-Matching Network for Video Object Segmentation

By Xiaohui Zeng, Renjie Liao, Li Gu, Yuwen Xiong, Sanja Fidler, Raquel Urtasun. 2019

77 citations recorded by the source corpus.

In this paper, we propose the differentiable mask-matching network (DMM-Net) for solving the video object segmentation problem where the initial object masks are provided. Relying on the Mask R-CNN backbone, we extract mask proposals per frame and formulate the matching between object templates and proposals as a linear assignment problem where thA heading inside a blocke cost matrix is predicted by a deep convolutional neural network. We propose a differentiable matching layer which unrolls a projected gradient descent algorithm in which the projection step exploits the Dykstra's algorithm. We prove that under mild conditions, the matching is guaranteed to converge to the optimal one. In practice, it achieves similar performance compared to the Hungarian algorithm during inference. Meanwhile, we can back-propagate through it to learn the cost matrix. After matching, a U-Net style architecture is exploited to refine the matched mask per time step. On DAVIS 2017 dataset, DMM-Net achieves the best performance without online learning on the first frames and the 2nd best with it. Without any fine-tuning, DMM-Net performs comparably to state-of-the-art methods on SegTrack v2 dataset. At last, our differentiable matching layer is very simple to implement; we attach the PyTorch code in the supplementary material which is less than $50$ lines long.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Object-Centric Learning with Slot Attention (2020, background): “We leave the exploration of other matching algorithms [36, 37] for future work.”
SwiftNet: Real-time Video Object Segmentation (2021, methodology): “For the former one, [35, 34, 42, 43, 14, 29, 10] utilize only the first and/or last frame as reference, demonstrate favorable segmentation speed but suffer uncompetitive accuracy due to inadequate modeling over object variation.”
STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videos (2020, methodology): “Current stateof-the-art methods for this task are either proposal-based [105,107] or focus on foreground-background segmentation [36,40,77,78,82,83,102,103].”
Video Object Segmentation with Episodic Graph Memory Networks (2020, methodology): “MSK OSMN RGMP OnAVOS RVOS OSVOS S2S AGAME PreMVOS DMMNet STM Method [33] [66] [59] [50] [48] [5] [64] [17] [26] [70] [31] Ours”
Delving into the Cyclic Mechanism in Semi-supervised Video Object Segmentation (2020, methodology): “05 DMM-Net [8] X X 60.”