FusionSeg: Learning to Combine Motion and Appearance for Fully Automatic Segmentation of Generic Objects in Videos

By Suyog Dutt Jain, Bo Xiong, Kristen Grauman. Computer Vision and Pattern Recognition, 2017

398 citations recorded by the source corpus.

We propose an end-to-end learning framework for segmenting generic objects in videos. Our method learns to combine appearance and motion information to produce pixel level segmentation masks for all prominent objects in videos. We formulate this task as a structured prediction problem and design a two-stream fully convolutional neural network which fuses together motion and appearance in a unified framework. Since large-scale video datasets with pixel level segmentations are problematic, we show how to bootstrap weakly annotated videos together with existing image recognition datasets for training. Through experiments on three challenging video segmentation benchmarks, our method substantially improves the state-of-the-art for segmenting generic (unseen) objects. Code and pre-trained models are available on the project website.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/90034490abbbeaff8b9e1d602a64d69215600244

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Online Adaptation of Convolutional Neural Networks for Video Object Segmentation (2017, methodology): “[20], the resulting objectness network alone already performs well on DAVIS, but here we use objectness only as a pretraining step.”
PoTion: Pose MoTion Representation for Action Recognition (2018, background): “In particular, two-stream approaches have shown promising results in different video understanding tasks such as video classification [12, 32, 43], action localization [20, 30] and video segmentation [18, 38].”
Video Instance Segmentation (2019, background): “In unsupervised scenario, a single foreground object is segmented [29, 14, 30].”
YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark (2018, methodology): “Although some recent approaches [11,10,13] have been proposed to leverage temporal consistency, they depend on models pretrained on other tasks such as optical flow [17,18] or motion segmentation [19], to extract temporal information.”
Pyramid Dilated Deeper ConvLSTM for Video Salient Object Detection (2018, background): “ARP*[20] LVO[34] FSEG[19] LMP[33] SFL*[5] FST*[30] SAGE*[42] Ours Ours+”
SwiftNet: Real-time Video Object Segmentation (2021, methodology): “For the former one, [35, 34, 42, 43, 14, 29, 10] utilize only the first and/or last frame as reference, demonstrate favorable segmentation speed but suffer uncompetitive accuracy due to inadequate modeling over object variation.”
MASK-RL: Multiagent Video Object Segmentation Framework Through Reinforcement Learning (2020, background): “…object segmentation methods typically pose the problem as the identiﬁcation of spatiotemporal “tubes” (representing motion trajectories) [4], [10], [16], [23], [26], [30], [31], [41], [43] by tracking points or patches over consecutive frames in order to localize areas where coherent object motion…”
An Efficient Optical Flow Based Motion Detection Method for Non-stationary Scenes (2018, background): “FSEG [6] 70.”
Learning Spherical Convolution for Fast Features from 360° Imagery (2017, background): “CNNs are responsible for state-of-the-art results across a wide range of vision problems, including image recognition [17, 42], object detection [12, 30], image and video segmentation [16, 21, 28], and action detection [10, 32].”
Instance Embedding Transfer to Unsupervised Video Object Segmentation (2018, methodology): “on image segmentation datasets and later adapt the network to the video domain, sometimes combined with optical flow [4, 6, 17, 37, 38, 41].”
YouTube-VOS: Sequence-to-Sequence Video Object Segmentation (2018, methodology): “temporal information. Even though simple, such an online learning or one-shot learning scheme achieves top performance on video object segmentation benchmarks [33,21]. Although some recent approaches [11,8,41] have been proposed to leverage temporal consistency, they depend on models pretrained on o”
Zero-Shot Video Object Segmentation via Attentive Graph Neural Networks (2019, methodology): “The correlation between two frames is efficiently captured by an attention mechanism, which avoids time-consuming optical flow estimation [7, 57, 21, 58, 31].”
Video Object Segmentation without Temporal Information (2017, neutral): “For context, we also add the results of the latest unsupervised techniques (ARP [30], FSEG [24], LMP [66],”
STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videos (2020, methodology): “Current stateof-the-art methods for this task are either proposal-based [105,107] or focus on foreground-background segmentation [36,40,77,78,82,83,102,103].”
AMP: Adaptive Masked Proxies for Few-Shot Segmentation (2019, methodology): “Measure FSeg [10] LVO [36] MOTAdapt [30] ARP [13] PDB [33] AMP + CRF (Ours) J Mean 70.7 75.9 77.2 76.2 77.2 78.9 Recall 83.5 89.1 87.8 91.1 90.1 91.6 Decay 1.5 7.0 5.0 7.0 0.9 4.7 F Mean 65.3 72.1 77.4 70.6 74.5 78.4 Recall 73.8 83.4 84.4 83.5 84.4 87.3 Decay 1.8 1.3 3.3 7.9 0.2 2.7 Table 6: Quantit”
Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation (2019, background): “evel instance segmentations rather than frame-level bounding boxes. Video Object Segmentation. The task of video object segmentation requires segmenting foreground objects in a class-agnostic fashion [43,21,36,38], often by leveraging ground truth masks available for the ﬁrst frame during inference ”
Multi-Stream Deep Neural Networks for RGB-D Egocentric Action Recognition (2019, background): “, action recognition [14], video-based facial landmark detection [88] and object segmentation [89]) have benefited from optical flow.”
Anabranch network for camouflaged object segmentation (2019, background): “There are several works that use two-stream architectures (Simonyan and Zisserman, 2014a; Feichtenhofer et al., 2016; Wang et al., 2016; Jain et al., 2017; He et al., 2017; Li et al., 2017b).”