A Quantitative Analysis of Current Practices in Optical Flow Estimation and the Principles Behind Them

By Deqing Sun, Stefan Roth, Michael J. Black. 2013

693 citations recorded by the source corpus.

The accuracy of optical flow estimation algorithms has been improving steadily as evidenced by results on the Middlebury optical flow benchmark. The typical formulation, however, has changed little since the work of Horn and Schunck. We attempt to uncover what has made recent advances possible through a thorough analysis of how the objective function, the optimization method, and modern implementation practices influence accuracy. We discover that “classical” flow formulations perform surprisingly well when combined with modern optimization and implementation techniques. One key implementation detail is the median filtering of intermediate flow fields during optimization. While this improves the robustness of classical methods it actually leads to higher energy solutions, meaning that these methods are not optimizing the original objective function. To understand the principles behind this phenomenon, we derive a new objective function that formalizes the median filtering heuristic. This objective function includes a non-local smoothness term that robustly integrates flow estimates over large spatial neighborhoods. By modifying this new term to include information about flow and image boundaries we develop a method that can better preserve motion details. To take advantage of the trend towards video in wide-screen format, we further introduce an asymmetric pyramid downsampling scheme that enables the estimation of longer range horizontal motions. The methods are evaluated on the Middlebury, MPI Sintel, and KITTI datasets using the same parameter settings.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Lightweight Probabilistic Deep Networks (2018, background): “ods have been proposed that typically exploit the brightness constancy assumption [49], Dosovitskiy et al.”
Optical Flow Estimation Using a Spatial Pyramid Network (2016, methodology): “As is standard, with classical formulations [36], we warp one image towards the other using the current ﬂow, and repeat this process at each pyramid level.”
Attention-Guided Network for Ghost-Free High Dynamic Range Imaging (2019, methodology): “Furthermore, we also apply the proposed AHDRNet with the input images aligned via estimated optical flow [32] (referred to as Ours+OF).”
Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring (2016, neutral): “CNN based approaches are not exception [25, 26].”
Pattern Recognition (2015, critical): “In the real world applications, the computation time of the traditional optical ﬂow methods [20,21] is unacceptable.”
Computer Vision for Autonomous Vehicles: Problems, Datasets and State-of-the-Art (2017, methodology): “A detailed study of optical flow methods is provided by Sun et al. (2014).”
COTR: Correspondence Transformer for Matching Across Images (2021, background): “The other is to interpret correspondence as a dense process, where every pixel in the first image maps to a pixel in the second image [30, 57, 72, 67].”
Unsupervised convolutional neural networks for motion estimation (2016, background): “Finally, in Section 4 we give some conclusion.”
Back to Basics: Unsupervised Learning of Optical Flow via Brightness Constancy and Motion Smoothness (2016, background): “Recovering optical flow between two frames is a well studied problem, with much previous work founded on variational formulations [7, 2, 13, 12].”
Unsupervised Deep Learning for Optical Flow Estimation (2017, methodology): “However, accurate estimation of optical flow remains a challenging problem (Sun, Roth, and Black 2014; Butler et al. 2012).”
Layered RGBD scene flow estimation (2015, methodology): “Teddy Cones RMS AAE RMS AAE Proposed (K = 4) 0.09 0.17 0.12 0.13 SRSF [27] 0.49 0.46 0.45 0.37 Hadfield and Bowden [17] 0.52 1.36 0.59 1.61 Basha et al. [7] (2 views) 0.57 1.01 0.58 0.39 Quiroga et al. [28] 0.94 0.84 0.79 0.52 FC-2Layers-FF [41]∗ 1.90 0.24 2.80 0.18 Classic+NLP [38]∗ 2.00 0.24 2.79 ”
SegFlow: Joint Learning for Video Object Segmentation and Optical Flow (2017, background): “For instance, when an object moves fast, the optical flow methods [2, 5, 37] are not effective in capturing the movement and hence generate incomplete flow within the object (see Figure 4 for an example).”
Video-Based Facial Micro-Expression Analysis: A Survey of Datasets, Features and Algorithms (2021, background): “Xu et al. [84] proposed facial dynamics map (FDM) to capture small facial motions based on optical ﬂow estimation [72].”
UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss (2017, background): “…data loss is now deﬁned as where f D ( I 1 ( x ) , I 2 ( x (cid:48) )) measures the photometric difference between two putatively corresponding pixels x and x (cid:48) , and ρ ( x ) = ( x 2 + 2 ) γ is the robust generalized Charbonnier penalty function (Sun, Roth, and Black 2014), with γ = 0 .”
Mnemonic Descent Method: A Recurrent Process Applied for End-to-End Face Alignment (2016, background): “Non-linear least squares optimisation problems often arise in computer vision, including but not limited to Structure-from-Motion [9, 23], rigid and deformable image alignment [33, 35], optical flow estimation [33, 66, 46, 45], CNN RNN”
Efficient sparse-to-dense optical flow estimation using a learned basis and layers (2015, methodology): “On Sintel it is more accurate than recent methods like MDP-Flow2 [53], EPPM [4], MLDP-OF [33], Classic+NLP [44] and the traditional layered approach, FC-2Layers-FF [46], which is a least two orders of magnitude slower.”