PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume

By Deqing Sun, Xiaodong Yang, Ming-Yu Liu, Jan Kautz. 2017

2,865 citations recorded by the source corpus.

We present a compact but effective CNN model for optical flow, called PWC-Net. PWC-Net has been designed according to simple and well-established principles: pyramidal processing, warping, and the use of a cost volume. Cast in a learnable feature pyramid, PWC-Net uses the current optical flow estimate to warp the CNN features of the second image. It then uses the warped features and features of the first image to construct a cost volume, which is processed by a CNN to estimate the optical flow. PWC-Net is 17 times smaller in size and easier to train than the recent FlowNet2 model. Moreover, it outperforms all published optical flow methods on the MPI Sintel final pass and KITTI 2015 benchmarks, running at about 35 fps on Sintel resolution (1024 × 436) images. Our models are available on our project website.

Subfields: Computer Vision and Pattern Recognition, Media Technology, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Ming-Hsuan Yang
3. Dacheng Tao
4. Xiaogang Wang
5. Jitendra Malik
6. Andrew Zisserman
7. Andrea Vedaldi
8. Trevor Darrell

In Media Technology

1. Radu Timofte
2. Ming-Hsuan Yang
3. Luc Van Gool
4. Chen Change Loy
5. Chao Dong
6. Lei Zhang
7. Wangmeng Zuo
8. Kai Zhang

In Model Architectures

1. Chunhua Shen
2. Luc Van Gool
3. Philip H. S. Torr
4. Vladlen Koltun
5. Raquel Urtasun
6. Ming-Hsuan Yang
7. Dacheng Tao
8. Chen Change Loy

Citations in the corpus

• Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, methodology): “On Sintel [10], our results are slightly better than RAFT on Sintel and outperform PWCNet on KITTI [52].”
• Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer (2019, methodology): “The ground truth (obtained from stereo matching [42]) is shown for reference.”
• Pyramid Stereo Matching Network (2018, methodology): “Net [28] improves optical flow estimation by using feature pyramids.”
• DeepFakes and Beyond: A Survey of Face Manipulation and Fake Detection (2020, methodology): “The PWC-Net approach proposed in [150] was considered to compute the flow from original to manipulated and vice versa.”
• MARS: Motion-Augmented RGB Stream for Action Recognition (2019, methodology): “We also evaluate the performance of two computationally efficient flow approaches, MPEGFlow [17] and PWC-Net [34] on MiniKinetics.”
• AutoFlow: Learning a Better Training Set for Optical Flow (2021, methodology): “To increase the diversity of the training data, we apply data augmentation [8, 45] to the rendered data.”
• Channel Attention Is All You Need for Video Frame Interpolation (2020, methodology): “We also use recent PWC-Net (Sun et al. 2018) to compute bi-directional optical flow and use an occlusion reasoning algorithm (Baker et al. 2010) to construct a strong baseline for two-step frame interpolation algorithm for comparison.”
• Deep Image Deblurring: A Survey (2022, methodology): “Pan et al. (2020) achieve state-of-the-art performance on the DVD dataset by using a cascaded network and PWC-Net to estimate optical flow for calculating sharpness priors Sun et al. (2018).”
• Opening up Open World Tracking (2021, methodology): “To compare performance, we use top-1 accuracy: accuracy = number of successful associations total number of paired GTs For methods using optical flow in Table 3, we use PWC-Net [74] (with fine-tuned weights on MPI Sintel [12]) to calculate optical flow vectors.”
• Deepfake Video Detection through Optical Flow Based CNN (2019, methodology): “So, for this reason, for each frame f(t), at a certain time t, a forward flow OF(f(t), f(t + 1)) is extracted using the CNN model for optical flow called PWC-Net [13].”
• Conditional Affordance Learning for Driving in Urban Environments (2018, background): “, for object detection [8, 9], image segmentation [10, 11], depth [12, 13] or motion estimation [14, 15].”
• Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers (2021, background): “For short-range correspondences across time, optical flow algorithms [30, 66, 70] are highly effective.”
• PolyTransform: Deep Polygon Transformer for Instance Segmentation (2019, methodology): “1) We replace PolyTransform with a refinement network inspired by DeepLabV3 [7] and PWC-Net [59] .”
• Learning Depth with Convolutional Spatial Propagation Network (2018, methodology): “PWCNet [85] follows a similar idea while having cost volumes calculated using a”
• Single-Stage Semantic Segmentation From Image Labels (2020, methodology): “The recent Frame-to-Frame system [31] is also supervised with saliency and is trained on additional 15K images mined from videos, which requires state-of-the-art optical flow [50].”
• DeepV2D: Video to Depth with Differentiable Structure from Motion (2018, methodology): “PWCNet+Ceres takes the output of an optical flow network, PWCNet (Sun et al., 2018), and performs joint optimization of depth and pose using the Ceres solver (Agarwal et al.”