SegFlow: Joint Learning for Video Object Segmentation and Optical Flow

By Jingchun Cheng, Yi-Hsuan Tsai, Shengjin Wang, Ming-Hsuan Yang. IEEE International Conference on Computer Vision, 2017

431 citations recorded by the source corpus.

This paper proposes an end-to-end trainable network, SegFlow, for simultaneously predicting pixel-wise object segmentation and optical flow in videos. The proposed SegFlow has two branches where useful information of object segmentation and optical flow is propagated bidirectionally in a unified framework. The segmentation branch is based on a fully convolutional network, which has been proved effective in image segmentation task, and the optical flow branch takes advantage of the FlowNet model. The unified framework is trained iteratively offline to learn a generic notion, and fine-tuned online for specific objects. Extensive experiments on both the video object segmentation and optical flow datasets demonstrate that introducing optical flow improves the performance of segmentation and vice versa, against the state-of-the-art algorithms.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/ab494d4f2e8f4efb830f15d78cd68ef7f355c76b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Channel Attention Is All You Need for Video Frame Interpolation (2020, background): “2018), object detection and segmentation (Cheng et al. 2017; Zhu et al. 2018; 2017), video prediction (Li et al.”
Instance Embedding Transfer to Unsupervised Video Object Segmentation (2018, methodology): “The advantage of combining motion and objectness is reported in several previous methods as well [6, 16, 37].”
VideoMatch: Matching based Video Object Segmentation (2018, background): “We also compare to OSVOS [5], MSK [25], OnAVOS [53] and SFL [8], disabling their fine-tuning step.”
DSNet: Joint Learning for Scene Segmentation and Disparity Estimation (2019, methodology): “The current works [18] [10] have opened a way to the fusion of networks for deep semantic and deep optical flow.”
Fast Video Object Segmentation by Reference-Guided Mask Propagation (2018, neutral): “3s SegFlow [7] 74.”
Occlusions, Motion and Depth Boundaries with a Generic Network for Disparity, Optical Flow or Scene Flow Estimation (2018, background): “Most of these assume a precomputed optical flow, and only few estimate the segmentation and the flow jointly [40, 8].”
The 2019 DAVIS Challenge on VOS: Unsupervised Multi-Object Segmentation (2019, background): “The availability of such dataset introduced the first wave of deep learning based methods in the field [4], [5], [6], [7], [8], [9], [10], [11], [12], [13] that considerably improved the state of the art.”
DF-Net: Unsupervised Joint Learning of Depth and Flow using Cross-Task Consistency (2018, background): “For examples, joint learning of video segmentation and optical flow through layered models [6,56] or feature sharing [9] helps improve accuracy at motion boundaries.”
Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation (2019, background): “Still, the efficient test-time inference has led to their widespread adoption as a sub-module in applications requiring to process temporal information, including video object segmentation [13], video recognition [15, 43, 64], and video style transfer [11].”
SG-One: Similarity Guidance Network for One-Shot Semantic Segmentation (2018, background): “SegFlow [39] leverages the optical flow of moving objects to assist the segmentation process.”
Video Instance Segmentation (2019, background): “Visual similarity [3, 5, 31], motion cues [6], and temporal consistency [21, 35] are extracted to identity the same object across the video.”
Video Object Segmentation with Episodic Graph Memory Networks (2020, methodology): “LTV FST COSEG ARP LVO PDB FSEG SFL MotAdapt LSMO AGS COSNet AGNN Method [30] [32] [47] [19] [45] [42] [16] [8] [41] [46] [55] [24] [52] Ours”
YouTube-VOS: Sequence-to-Sequence Video Object Segmentation (2018, methodology): “Video object segmentation, which targets at segmenting a particular object instance throughout the entire video sequence given only the object mask on the first frame, has attracted much attention from the vision community recently [6,32,50,8,11,22,41,19,44].”
See More, Know More: Unsupervised Video Object Segmentation With Co-Attention Siamese Networks (2019, methodology): “These methods compute optical flows across a few consecutive frames [53, 24, 9, 32, 33], which is limited to a local receptive window in the temporal domain.”
Learning Video Object Segmentation From Unlabeled Videos (2020, methodology): “As our method does not use any first-frame fine-tuning [6, 35] or online learning [55] technique, it is fast for inference.”
SAIL-VOS: Semantic Amodal Instance Level Video Object Segmentation – A Synthetic Dataset and Baselines (2019, background): “A variety of methods have tackled the unsupervised ( i.e ., no object annotation available during testing) [73, 71, 27, 44, 95, 96, 48] and the semi-supervised ( i.e ., ﬁrst frame annotation available during testing) [12, 54, 49, 11, 53, 98, 45, 18, 63, 105, 17, 15, 102, 100] setting.”
Zero-Shot Video Object Segmentation via Attentive Graph Neural Networks (2019, methodology): “The correlation between two frames is efficiently captured by an attention mechanism, which avoids time-consuming optical flow estimation [7, 57, 21, 58, 31].”
Learning Unsupervised Video Object Segmentation Through Visual Attention (2019, methodology): “Later, with the renaissance of neural network, many deep learning based models were proposed, which typically use multilayer perceptron based moving objectness detector, adopt two-stream architecture [67, 33], or CNN encoder-decoder structure [66, 11, 45, 46, 64].”