Learning Motion Patterns in Videos

By Pavel Tokmakov, Alahari Karteek, Cordelia Schmid. 2016

278 citations recorded by the source corpus.

The problem of determining whether an object is in motion, irrespective of camera motion, is far from being solved. We address this challenging task by learning motion patterns in videos. The core of our approach is a fully convolutional network, which is learned entirely from synthetic video sequences, and their ground-truth optical flow and motion segmentation. This encoder-decoder style architecture first learns a coarse representation of the optical flow field features, and then refines it iteratively to produce motion labels at the original high-resolution. We further improve this labeling with an objectness map and a conditional random field, to account for errors in optical flow, and also to focus on moving things rather than stuff. The output label of each pixel denotes whether it has undergone independent motion, i.e., irrespective of camera motion. We demonstrate the benefits of this learning framework on the moving object segmentation task, where the goal is to segment all objects in motion. Our approach outperforms the top method on the recently released DAVIS benchmark dataset, comprising real-world sequences, by 5.6%. We also evaluate on the Berkeley motion segmentation database, achieving state-of-the-art results.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Occluded Video Instance Segmentation: A Benchmark (2021, background): “Unsupervised VOS methods [33,56,50] aim to segment the primary objects in a video without the first frame annotations.”
YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark (2018, background): “Although some recent approaches [11,10,13] have been proposed to leverage temporal consistency, they depend on models pretrained on other tasks such as optical ﬂow [17,18] or motion segmentation [19], to extract temporal information.”
Video Instance Segmentation (2019, background): “isual similarity [3,5], motion cues [6], and temporal consistency [19,32] are extracted to identity the same object across the video. In unsupervised scenario, a single foreground object is segmented [27,13,28]. In both settings, algorithms consider the target objects as general objects and does not”
AMP: Adaptive Masked Proxies for Few-Shot Segmentation (2019, unknown): “Measure FST [21] CVOS [34] CUT [11] MPNet-V[35] LVO[36] MotAdapt [30] AMP (ours) P 76.”
A Survey on Deep Learning Technique for Video Segmentation (2021, methodology): “Core Architecture InstanceFlow Training Dataset 2017 FSEG [66] CVPR Two-Stream FCN Object ✓ ImageNet VID [67] + DAVIS16 [17] SFL [68] ICCV Two-Stream FCN Object ✓ DAVIS16 [17] LVO [69] ICCV Two-Stream FCN Object ✓ DAVIS16 [17] LMP [70] ICCV FCN Object ✓ FT3D [71] NRF [72] ICCV FCN Object ✓ Youtube-O”
DMM-Net: Differentiable Mask-Matching Network for Video Object Segmentation (2019, background): “Methods under the unsupervised category [5, 12, 42, 34] typically exploit the dense optical flow and appearance feature to group the pixels within the spatio-temporal neighborhood.”
Pyramid Dilated Deeper ConvLSTM for Video Salient Object Detection (2018, background): “ARP*[20] LVO[34] FSEG[19] LMP[33] SFL*[5] FST*[30] SAGE*[42] Ours Ours+”
Online Adaptation of Convolutional Neural Networks for Video Object Segmentation (2017, methodology): “Furthermore, optical ﬂow has been used as an additional temporal cue in conjunction with deep learning in the semi-supervised [24, 35] and unsupervised setting [40], in which the ground truth for the ﬁrst frame is not available.”
Learning Unsupervised Video Object Segmentation Through Visual Attention (2019, methodology): “Later, with the renaissance of neural network, many deep learning based models were proposed, which typically use multilayer perceptron based moving objectness detector, adopt two-stream architecture [67, 33], or CNN encoder-decoder structure [66, 11, 45, 46, 64].”
Video Object Segmentation with Episodic Graph Memory Networks (2020, background): “Table 3. Evaluation of Z-VOS on DAVIS 16 val set[32] (x4.2), with region similarity J, boundary accuracy F and time stability T . MSG NLC CUT FST SFL LMP FSEG LVO UOVOS Method [27] [11] [17] [30] [7] [41] [15] [42] [66] J Mean " 53.3 55.1 55.2 55.8 64.7 70.0 70.7 75.9 73.9 Recall" 61.6 55.”
VideoMatch: Matching based Video Object Segmentation (2018, background): “More recently, deep net based approaches have been discussed [47, 48, 22].”
YouTube-VOS: Sequence-to-Sequence Video Object Segmentation (2018, background): “However, due to the lack of training videos, they use a pretrained motion segmentation model [40] and optical-flow model [20], which leads to suboptimal results since the model is not trained end-to-end to best capture spatial-temporal features.”
An Efficient Optical Flow Based Motion Detection Method for Non-stationary Scenes (2018, background): “al [4], Mennatullah Siam et.”
Visual SLAM for Automated Driving: Exploring the Applications of Deep Learning (2018, methodology): “A recent trend [56, 32, 17, 61, 24] for learning motion in videos has emerged.”
Video Object Segmentation using Teacher-Student Adaptation in a Human Robot Interaction (HRI) Setting (2018, background): “unsupervised methods [14][30][9][29] attempt to discover the primary object automatically and segment it through the video sequence.”
MODNet: Motion and Appearance based Moving Object Detection Network for Autonomous Driving (2018, methodology): “Table IV shows that our method outperforms the state of the art on DAVIS in unsupervised motion segmentation, except for MPNet+CRF.”
The 2019 DAVIS Challenge on VOS: Unsupervised Multi-Object Segmentation (2019, background): “The availability of such dataset introduced the ﬁrst wave of deep learning based methods in the ﬁeld [4], [5], [6], [7], [8], [9], [10], [11], [12], [13] that considerably improved the state of the art.”
PoTion: Pose MoTion Representation for Action Recognition (2018, unknown): “In particular, two-stream approaches have shown promising results in different video understanding tasks such as video classification [12, 32, 43], action localization [20, 30] and video segmentation [18, 38].”
Video Object Segmentation without Temporal Information (2017, unknown): “For context, we also add the results of the latest unsupervised techniques (ARP [30], FSEG [24], LMP [66],”
Instance Embedding Transfer to Unsupervised Video Object Segmentation (2018, methodology): “For example, LMP [36] trains a network which takes optical flow as an input to separate moving and non-moving regions and then combines the results with objectness cues from SharpMask [34] to generate the moving object segmentation.”