A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation

By Nikolaus Mayer, Eddy Ilg, Philip Häusser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, Thomas Brox. 2015

3,047 citations recorded by the source corpus.

Recent work has shown that optical flow estimation can be formulated as a supervised learning task and can be successfully solved with convolutional networks. Training of the so-called FlowNet was enabled by a large synthetically generated dataset. The present paper extends the concept of optical flow estimation via convolutional networks to disparity and scene flow estimation. To this end, we propose three synthetic stereo video datasets with sufficient realism, variation, and size to successfully train large networks. Our datasets are the first large-scale datasets to enable training and evaluation of scene flow methods. Besides the datasets, we present a convolutional network for real-time disparity estimation that provides state-of-the-art results. By combining a flow and disparity estimation network and training it jointly, we demonstrate the first scene flow estimation with a convolutional network.

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

IRS: A Large Synthetic Indoor Robotics Stereo Dataset for Disparity and Surface Normal Estimation (2019, methodology): “It has been shown that the existing state-of-the-art deep models [4], [15] trained on these synthetic datasets did not work well on the complicated real-world indoor scenes.”
RayNet: Learning Volumetric 3D Reconstruction with Ray Potentials (2018, background): “In contrast, recent learning-based solutions to motion estimation [10, 15, 24], stereo matching [20, 21, 42] and 3D reconstruction [5, 6, 9, 16, 37] have demonstrated impressive results by learning feature representations that are much more robust to local viewpoint and lighting changes.”
Learning a Multi-View Stereo Machine (2017, background): “Most recently, CNN based learning methods have been proposed for 3D reconstruction by learning image patch similarity functions [60, 18, 23] and end-to-end disparity regression from stereo pairs [37, 28].”
FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks (2016, methodology): “Because noise tends to be a problem with small displacements, we add convolutions between the upconvolutions to obtain smoother estimates like in [18].”
Playing for Data: Ground Truth from Computer Games (2016, background): “Renderings of entire synthetic environments have been proposed for training convolutional networks for stereo disparity and scene flow estimation [31].”
Playing for Benchmarks (2017, methodology): “We sampled 500 random images each from VIPER, SYNTHIA [55], Virtual KITTI [17], the Frei-burg driving sequence [41], and the Urban Canyon dataset (regular perspective camera) [64], as well as Cityscapes as a real-world reference.”
Domain Adaptation for Visual Applications: A Comprehensive Survey (2017, background): “Such virtually generated and controlled environments come with different levels of labeling for free and therefore have great promise for deep learning across a variety of computer vision problems, including optical flow [179, 180, 181, 182], object trackers [183, 177], depth estimation from RGB [18”
Optical Flow Estimation Using a Spatial Pyramid Network (2016, methodology): “Here we also fine-tune the model using Driving and Monkaa scenes from [30] and evaluate the fine-tuned model SPyNet+ft*.”
Pyramid Stereo Matching Network (2018, methodology): “Both DispNet [19] and CRL [21] reuse hierarchical information, concatenating features from lower layers with those from higher layers.”
Unifying Flow, Stereo and Depth Estimation (2022, methodology): “Following RAFT [21], we further finetune our Things trained model on several mixed datasets that consist of KITTI 2015 [14], HD1K [108], FlyingThings3D [58] and Sintel [13] training sets.”
Learning Motion Patterns in Videos (2016, methodology): “We adopt the popular approach of using synthetic datasets, followed in other work [8, 11, 23].”
Real-Time Self-Adaptive Deep Stereo (2018, methodology): “Thereby, [21] ignited the spread of end-to-end stereo architectures [17, 24, 19, 4, 16, 11] that quickly outmatched any other technique on the KITTI benchmarks by leveraging on a peculiar training protocol.”
The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth (2021, result): “[61, 77, 55], [45] showed that integrating a plane-sweep stereo cost volume significantly improved results.”
Single-View View Synthesis With Multiplane Images (2020, methodology): “We use a DispNet-style network [20], specified in Table 1.”