AutoFlow: Learning a Better Training Set for Optical Flow

By Deqing Sun, Daniel Vlasic, Charles Herrmann, Varun Jampani, Michael Krainin, Huiwen Chang, Ramin Zabih, William T. Freeman, Ce Liu. 2021

137 citations recorded by the source corpus.

Synthetic datasets play a critical role in pre-training CNN models for optical flow, but they are painstaking to generate and hard to adapt to new applications. To automate the process, we present AutoFlow, a simple and effective method to render training data for optical flow that optimizes the performance of a model on a target dataset. AutoFlow takes a layered approach to render synthetic data, where the motion, shape, and appearance of each layer are controlled by learnable hyperparameters. Experimental results show that AutoFlow achieves state-of-the-art accuracy in pre-training both PWC-Net and RAFT. Our code and data are available at autoflow-google.github.io.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, methodology): “In all cases, we train on the AutoFlow dataset [74], which consists of 400, 000 image pairs, for 480 epochs using a cosine learning rate schedule which starts at a learning rate of 4e-4.”
Practical Stereo Matching via Cascaded Recurrent Network with Adaptive Correlation (2022, methodology): “Autoflow [40] introduced a simple method to render randomized polygons with motion for optical flow training.”
TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement (2023, background): “The four ResNet layers consist of feature dimensions of [64, 128, 256, 256], with 2 ResNet blocks per layer.”
GMFlow: Learning Optical Flow via Global Matching (2021, methodology): “, Virtual KITTI [5, 11], VIPER [33], REFRESH [26], AutoFlow [35] and TartanAir [44], they can be used to enhance Transformer’s generalization ability.”
Depth Pro: Sharp Monocular Metric Depth in Less Than a Second (2024, unknown): “(Note that this inverts the common practice of first training on synthetic data and then fine-tuning on real data (Gaidon et al., 2016; G ´ omez et al., 2023; Sun et al., 2021).)”
TAP-Vid: A Benchmark for Tracking Any Point in a Video (2022, unknown): “Optical Flow estimates dense motion, but only between image pairs [22, 40]; modern approaches [13, 25, 51, 63, 66] train and evaluate on synthetic scenes [6, 13, 46, 62], though depth scanners provide limited real data [16].”
PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking (2023, background): “We argue that long-range point tracking should not be treated as an extension of optical flow, where naturalism might indeed be discarded without ill effect [50].”
CoTracker: It is Better to Track Together (2023, background): “Trackers and optical flow estimators are often trained us-ing synthetic datasets [5, 12, 24, 30, 40, 52], as annotating real data can be challenging.”
Unifying Flow, Stereo and Depth Estimation (2022, background): “Further advances of network architectures and training strategies [16], [17], [21], [31], [43], [44], [45], [46], [47], [48], [49], [50], [51], [52], [53] have led to steady progress for learning-based methods, which today outperform traditional approaches by a large margin and are currently dominat”
FlowFormer: A Transformer Architecture for Optical Flow (2022, background): “Autoflow [41] is a dataset that provides training data covering various challenging visual disturbance, but its training code is not released yet.”
Synthetic Data from Diffusion Models Improves ImageNet Classification (2023, methodology): “Examples include tasks like semantic image segmentation [3, 10, 36, 37, 54, 66], optical flow estimation [14, 32, 63], human motion understanding [19, 29, 38, 67], and other dense prediction tasks [3, 70].”