A Database and Evaluation Methodology for Optical Flow

By Simon Baker, Daniel Scharstein, J. P. Lewis, Stefan Roth, Michael J. Black, Richard Szeliski. 2007

2,924 citations recorded by the source corpus.

The quantitative evaluation of optical flow algorithms by Barron et al. (1994) led to significant advances in performance. The challenges for optical flow algorithms today go beyond the datasets and evaluation methods proposed in that paper. Instead, they center on problems associated with complex natural scenes, including nonrigid motion, real sensor noise, and motion discontinuities. We propose a new set of benchmarks and evaluation methods for the next generation of optical flow algorithms. To that end, we contribute four types of data to test different aspects of optical flow algorithms: (1) sequences with nonrigid motion where the ground-truth flow is determined by tracking hidden fluorescent texture, (2) realistic synthetic sequences, (3) high frame-rate video used to study interpolation error, and (4) modified stereo sequences of static scenes. In addition to the average angular error used by Barron et al., we compute the absolute flow endpoint error, measures for frame interpolation error, improved statistics, and results at motion discontinuities and in textureless regions. In October 2007, we published the performance of several well-known methods on a preliminary version of our data to establish the current state of the art. We also made the data freely available on the web at http://vision.middlebury.edu/flow/. Subsequently a number of researchers have uploaded their results to our website and published papers using the data. A significant improvement in performance has already been achieved. In this paper we analyze the results obtained to date and draw a large number of conclusions from them.

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Microsoft COCO: Common Objects in Context (2014, background): “Some notable datasets include the Middlebury datasets for stereo vision [20], multi-view stereo [36] and optical flow [21].”
A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation (2015, background): “The first significant efforts to create standard datasets were the Middlebury datasets for stereo disparity estimation [20] and optical flow estimation [1].”
FlowNet: Learning Optical Flow with Convolutional Networks (2015, methodology): “The Middlebury dataset [2] contains only 8 image pairs for training, with ground truth flows generated using four different techniques.”
Playing for Data: Ground Truth from Computer Games (2016, background): “The use of synthetic data has been particularly significant in evaluating optical flow estimation due to the difficulty of obtaining accurate ground-truth flow measurements for real-world scenes [6, 7, 18, 32].”
An Uncertain Future: Forecasting from Static Images Using Variational Autoencoders (2016, background): “The directions of the trajectories at each point in time are color-coded according to the square on the right [2].”
Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation (2019, methodology): “We encode 2D offset vectors into RGB values, same as [3].”
Unsupervised Learning of Edges (2015, methodology): “The standard colorcoding scheme for optical flow maps 2D flow vectors into a 3D color space by encoding flow orientation via hue and magnitude via saturation [4].”
Playing for Benchmarks (2017, background): “The Middlebury benchmarks for stereo, multi-view stereo, and optical ﬂow played a pivotal role in the development of low-level vision algorithms [4, 57, 58, 59].”
SIFT Flow: Dense Correspondence across Scenes and Its Applications (2011, methodology): “Inspired by the recent advances in image alignment and scene parsing, we propose SIFT flow to establish the correspondence between images across scenes.”
Spatio-temporal video autoencoder with differentiable memory (2015, methodology): “The optical ﬂow throughout this section is displayed using the colour code from Baker et al. (2011), illustrated in Figure 2, top-right corner.”
Dense Optical Flow Prediction from a Static Image (2015, neutral): “1 ar X iv :1 50 5.”
Optical Flow Estimation Using a Spatial Pyramid Network (2016, methodology): “We report similar performance as FlowNet on Flying Chairs and Sintel [11] but are signiﬁcantly more accurate than FlowNet on Middlebury [4] and KITTI [18] after ﬁne tuning.”
MOT16: A Benchmark for Multi-Object Tracking (2016, background): “In the recent past, the computer vision community has developed centralized benchmarks for numerous tasks including object detection [14], pedestrian detection [12], 3D reconstruction [39], optical flow [7], [19], visual odometry [19], singleobject short-term tracking [26], and stereo estimation [19”
Video Frame Interpolation via Adaptive Separable Convolution (2017, methodology): “Common frame interpolation approaches estimate dense motion, typically optical ﬂow, between two input frames and then interpolate one or more intermediate frames guided by the motion [3, 53, 60].”
Video Object Segmentation using Teacher-Student Adaptation in a Human Robot Interaction (HRI) Setting (2018, methodology): “The input to the motion stream is the optical flow computed using [16], and converted into RGB representation using the Sintel color wheel representation [1].”
PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking (2023, background): “For many years, the Middlebury dataset [6] was the primary benchmark for stereo and motion estimation methods.”
Computer Vision for Autonomous Vehicles: Problems, Datasets and State-of-the-Art (2017, methodology): “(2012b); Scharstein & Szeliski (2002); Baker et al. (2011); Everingham et al.”