Optical Flow with Semantic Segmentation and Localized Layers

By Laura Sevilla-Lara, Deqing Sun, Varun Jampani, Michael J. Black. 2016

192 citations recorded by the source corpus.

Existing optical flow methods make generic, spatially homogeneous, assumptions about the spatial structure of the flow. In reality, optical flow varies across an image depending on object class. Simply put, different objects move differently. Here we exploit recent advances in static semantic scene segmentation to segment the image into objects of different types. We define different models of image motion in these regions depending on the type of object. For example, we model the motion on roads with homographies, vegetation with spatially smooth flow, and independently moving objects like cars and planes with affine motion plus deviations. We then pose the flow estimation problem using a novel formulation of localized layers, which addresses limitations of traditional layered models for dealing with complex scene motion. Our semantic flow method achieves the lowest error of any published monocular method in the KITTI-2015 flow benchmark and produces qualitatively better flow and segmentation than recent top methods on a wide range of natural videos.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs (2016, methodology): “Interestingly, the atrous convolution technique has also been adopted for a broader set of tasks, such as object detection [12], [77], instance-level segmentation [78], visual question answering [79], and optical flow [80].”
Kornia: an Open Source Differentiable Computer Vision Library for PyTorch (2019, background): “Multi-view reconstruction is a well understood problem with a good geometric model [21], and many approaches for matching and optimization [39, 8, 51], and some recent promising deep learning approaches [34].”
SegFlow: Joint Learning for Video Object Segmentation and Optical Flow (2017, background): “, video object segmentation and optical flow estimation, in that these two tasks have been known to be closely related to each other [41, 35].”
AutoScaler: Scale-Attention Networks for Visual Correspondence (2016, background): “Thus dense ﬂow approaches that exploit the semantics of the scene objects as well as the epipolar constraint would achieve signiﬁcant improvement [3, 45].”
Computer Vision for Autonomous Vehicles: Problems, Datasets and State-of-the-Art (2017, methodology): “When the goal is to represent real-world scenes with independent object motion as in scene flow or optical flow, superpixels can be generalized to rigidly moving segments (Vogel et al. (2015); Menze & Geiger (2015)), or semantic segments (Bai et al. (2016); Sevilla-Lara et al. (2016)).”
Occlusion Aware Unsupervised Learning of Optical Flow (2017, background): “The last group builds more sophisticated frameworks such as modeling depth or a layered representation of objects to reason about occlusion [50, 48, 58, 43].”
Deep Feature Flow for Video Recognition (2016, background): “Other works attempt to exploit semantic segmentation information to help optical flow estimation [37, 1, 19], e.”
GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose (2018, background): “[41] deﬁned different models of image motion according to semantic segmentation.”