Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation

By Junhwa Hur, Stefan Roth. Computer Vision and Pattern Recognition, 2019

299 citations recorded by the source corpus.

Deep learning approaches to optical flow estimation have seen rapid progress over the recent years. One common trait of many networks is that they refine an initial flow estimate either through multiple stages or across the levels of a coarse-to-fine representation. While leading to more accurate results, the downside of this is an increased number of parameters. Taking inspiration from both classical energy minimization approaches as well as residual networks, we propose an iterative residual refinement (IRR) scheme based on weight sharing that can be combined with several backbone networks. It reduces the number of parameters, improves the accuracy, or even achieves both. Moreover, we show that integrating occlusion prediction and bi-directional flow estimation into our IRR scheme can further boost the accuracy. Our full network achieves state-of-the-art results for both optical flow and occlusion estimation across several standard datasets.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/281c84088a4294bc0c74b4bb21d1800f2f925a5e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Luc Van Gool
3. Ming-Hsuan Yang
4. Xiaogang Wang
5. Jitendra Malik
6. Andrea Vedaldi
7. Andrew Zisserman
8. Trevor Darrell

In Media Technology

1. Jia Deng
2. Ce Liu
3. Deqing Sun
4. Zachary Teed
5. Dacheng Tao
6. Varun Jampani
7. Haofei Xu
8. Richard I. Hartley

Citations in the corpus

• AutoFlow: Learning a Better Training Set for Optical Flow (2021, methodology): “Quite a few new network architectures were proposed based on the PWC-Net framework [17, 51, 20, 52].”
• Learning to Estimate Hidden Motions with Global Motion Aggregation (2021, methodology): “The second major contribution is the use of a gated recurrent unit (GRU) decoder for iterative residual refinement [19].”
• Unifying Flow, Stereo and Depth Estimation (2022, methodology): “Further advances of network architectures and training strategies [16], [17], [21], [31], [43], [44], [45], [46] have led to steady progress for learning-based methods, which today outperform traditional approaches by a large margin and are currently dominating the benchmarks including Sintel [13] a”
• RAFT: Recurrent All-Pairs Field Transforms for Optical Flow (2020, methodology): “55 IRR-PWC [24] (1.”