SelFlow: Self-Supervised Learning of Optical Flow

By Pengpeng Liu, Michael R. Lyu, Irwin King, Jia Xu. 2019

340 citations recorded by the source corpus.

We present a self-supervised learning approach for optical flow. Our method distills reliable flow estimations from non-occluded pixels, and uses these predictions as ground truth to learn optical flow for hallucinated occlusions. We further design a simple CNN to utilize temporal information from multiple frames for better flow estimation. These two principles lead to an approach that yields the best performance for unsupervised optical flow learning on the challenging benchmarks including MPI Sintel, KITTI 2012 and 2015. More notably, our self-supervised pre-trained model provides an excellent initialization for supervised fine-tuning. Our fine-tuned models achieve state-of-the-art results on all three datasets. At the time of writing, we achieve EPE=4.26 on the Sintel benchmark, outperforming all submitted methods.

Subfields: Computer Vision and Pattern Recognition, Media Technology, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Ming-Hsuan Yang
3. Andrew Zisserman
4. Dacheng Tao
5. Andrea Vedaldi
6. Xiaogang Wang
7. Jitendra Malik
8. Trevor Darrell

In Media Technology

1. Ce Liu
2. Jia Deng
3. Deqing Sun
4. William T. Freeman
5. Hongdong Li
6. Ankush Gupta
7. Varun Jampani
8. Carl Doersch

In Representation & Self-Supervised Learning

1. Andrew Zisserman
2. Jia Deng
3. William T. Freeman
4. Ce Liu
5. Carl Doersch
6. Deqing Sun
7. Ankush Gupta
8. Charles Herrmann

Citations in the corpus

• CrossTransformers: spatially-aware few-shot transfer (2020, methodology): “Also particularly relevant are methods that use self-supervision for correspondence [41, 50, 90], which may in future work improve the correspondences that CrossTransformers use.”
• AutoFlow: Learning a Better Training Set for Optical Flow (2021, methodology): “, models pre-trained on FlyingChairs and FlyingThings [48, 51] are more accurate on Sintel than models [21, 28] trained on Sintel image pairs using self-supervised loss.”
• CoTracker3: Simpler and Better Point Tracking by Pseudo-Labeling Real Videos (2025, neutral): “Therefore, it usually requires multi-frame estimates [14], explicit reasoning about occlusions [42], hand-crafted loss terms [23, 25], or various data augmentation strategies [21].”
• RAFT: Recurrent All-Pairs Field Transforms for Optical Flow (2020, methodology): “Our method ranks 1st on both the Sintel clean and final passes, and outperforms the SelFlow[26], the best performing prior work, by 0.”
• Learning to Estimate Hidden Motions with Global Motion Aggregation (2021, background): “In the deep learning literature, the occlusion problem has been addressed in an unsupervised learning setting [41, 21, 25], however, existing supervised learning approaches all rely on convolutions to interpolate in occluded regions, which are prone to failure for more significant occlusions.”
• Self-Supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidance (2020, background): “Most existing works follow the classical way in considering optical flow, which can also be predicted in an unsupervised fashion [36,50].”