Flow Fields: Dense Correspondence Fields for Highly Accurate Large Displacement Optical Flow Estimation

By Christian Bailer, Bertram Taetz, Didier Stricker. 2015

224 citations recorded by the source corpus.

Modern large displacement optical flow algorithms usually use an initialization by either sparse descriptor matching techniques or dense approximate nearest neighbor fields. While the latter have the advantage of being dense, they have the major disadvantage of being very outlier prone as they are not designed to find the optical flow, but the visually most similar correspondence. In this paper we present a dense correspondence field approach that is much less outlier prone and thus much better suited for optical flow estimation than approximate nearest neighbor fields. Our approach is conceptually novel as it does not require explicit regularization, smoothing (like median filtering) or a new data term, but solely our novel purely data based search strategy that finds most inliers (even for small objects), while it effectively avoids finding outliers. Moreover, we present novel enhancements for outlier filtering. We show that our approach is better suited for large displacement optical flow estimation than state-of-the-art descriptor matching techniques. We do so by initializing EpicFlow (so far the best method on MPI-Sintel) with our Flow Fields instead of their originally used state-of-the-art descriptor matching technique. We significantly outperform the original EpicFlow on MPI-Sintel, KITTI and Middlebury.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks (2016, methodology): “Sintel: On Sintel, FlowNet2 consistently outperforms DeepFlow [31] and EpicFlow [21] and is on par with FlowFields [2].”
Playing for Benchmarks (2017, methodology): “FlowFields [3] 78.”
Video Frame Interpolation via Adaptive Separable Convolution (2017, methodology): “Our research is inspired by the success of applying deep learning to optical ﬂow estimation [2, 12, 16, 19, 50, 51, 52], artistic style transfer [17, 22, 28], and image enhancement [6, 9, 10, 46, 47, 55, 57, 62, 65].”
Learning Video Object Segmentation from Static Images (2016, methodology): “Given a video sequence, we compute the optical flow using EpicFlow [41] with Flow Fields matches [2] and convolutional boundaries [30].”
Fast Optical Flow Using Dense Inverse Search (2016, background): “EPE all s0-10 s10-40 s40+ time (s) CPU GPU FlowFields [16] 5.”
Hidden Two-Stream Convolutional Networks for Action Recognition (2017, methodology): “We explore both TV-L1 [24] and FlowFields [54].”
PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume (2017, unknown): “07% FlowFields [2] - - - - - 19.”
AutoScaler: Scale-Attention Networks for Visual Correspondence (2016, methodology): “Our relative large EPE-unmatched number is due to the simplicity of our interpolation scheme using Epicflow compared to the time-consuming comprehensive MRF post-processing step to propagate the estimated flow to occluded regions as employed by many competing systems such as Flowfields [4], Discrete”
Self-Supervised Monocular Depth Hints (2019, methodology): “[43] use added supervised loss [1] to solve for optical flow and Kuznietsov et al.”
DeMoN: Depth and Motion Network for Learning Monocular Stereo (2016, unknown): “We adopt the scaleinvariant error of [8], which is defined as sc-inv(z, ẑ) = √√√√ 1 n ∑ i d2i − 1n2 (∑ i di )2 , (8) with di = log zi−log ẑi.”
Deep End2End Voxel2Voxel Prediction (2015, background): “One can imagine extending V2V to have sparse inputs of color to make the problem well-posed for objects that can occur in various colors.”
Dynamic Video Segmentation Network (2018, background): “Optical flow estimation has been the subject of a number of research works [34, 33, 38, 39, 40, 41, 42].”
PatchBatch: A Batch Augmented Loss for Optical Flow (2015, methodology): “There are several significant differences between our work and [1].”
Occlusion Aware Unsupervised Learning of Optical Flow (2017, background): “Other trends include a coarse-to-fine estimation or a hierarchical framework to deal with large motion [13, 55, 15, 6], a design of loss penalty to improve the robustness to lighting change and motion blur [59, 46, 22, 54], and a more sophisticated framework to handle occlusion [2, 50] which we will”
Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation (2019, background): “Because occluded pixels do not have correspondences in the other frame, several approaches [5, 12, 16,23,32] aim to filter out these outliers to minimize their ill effects and ap ply post-processing to refine the estimates [22, 33, 48, 67].”
SDC – Stacked Dilated Convolution: A Unified Descriptor Network for Dense Matching Tasks (2019, methodology): “As mentioned in the main paper where we use triplet training with a thresholded hinge embedding loss [1], we have also experimented with the softmax loss of [8] and the PN loss of [3] (see Figure 6).”