Optical Flow Estimation Using a Spatial Pyramid Network

By Anurag Ranjan, Michael J. Black. 2016

1,419 citations recorded by the source corpus.

We learn to compute optical flow by combining a classical spatial-pyramid formulation with deep learning. This estimates large motions in a coarse-to-fine approach by warping one image of a pair at each pyramid level by the current flow estimate and computing an update to the flow. Instead of the standard minimization of an objective function at each pyramid level, we train one deep network per level to compute the flow update. Unlike the recent FlowNet approach, the networks do not need to deal with large motions, these are dealt with by the pyramid. This has several advantages. First, our Spatial Pyramid Network (SPyNet) is much simpler and 96% smaller than FlowNet in terms of model parameters. This makes it more efficient and appropriate for embedded applications. Second, since the flow at each pyramid level is small (

Subfields: Computer Vision and Pattern Recognition, Ophthalmology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Ophthalmology

Citations in the corpus

Beyond Skip Connections: Top-Down Modulation for Object Detection (2016, methodology): “Similar to the idea of refinement modules, Ranjan and Black [40] propose a spatial pyramid network, which computes a low resolution residual optical flow and iteratively propagates it to lower layers to obtain a high resolution output.”
RayNet: Learning Volumetric 3D Reconstruction with Ray Potentials (2018, background): “In contrast, recent learning-based solutions to motion estimation [10, 15, 24], stereo matching [20, 21, 42] and 3D reconstruction [5, 6, 9, 16, 37] have demonstrated impressive results by learning feature representations that are much more robust to local viewpoint and lighting changes.”
Hierarchical Surface Prediction for 3D Object Reconstruction (2017, background): “The benefits of using a coarse-to-fine approach in learning based optical flow prediction has also recently been pointed out [26], [15].”
FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks (2016, neutral): “Following FlowNet, several papers have studied optical flow estimation with CNNs: featuring a 3D CNN [30], unsupervised learning [1, 33], carefully designed rotationally invariant architectures [28], or a pyramidal approach based on the coarse-to-fine idea of variational methods [20].”
Video Frame Interpolation via Adaptive Separable Convolution (2017, methodology): “While this is larger than the recent AdaConv method [36], we plan to handle even larger motion by borrowing a multi-scale approach from optical ﬂow research [38].”
Pyramid Stereo Matching Network (2018, background): “SPyNet [23] introduces image pyramids to estimate optical flow in a coarse-to-fine approach.”
FlowFormer: A Transformer Architecture for Optical Flow (2022, methodology): “Then a series of works, represented by SpyNet [38], PWC-Net [43,44], LiteFlowNet [21,22] and VCN [55], employed coarse-to-fine and iterative estimation methodology.”
Context-Aware Synthesis for Video Frame Interpolation (2018, methodology): “In optical flow benchmarks, SPyNet performs less well than PWC-Net and as shown in Table 4, this decreased accuracy also affects the synthesis results.”
Video Frame Interpolation via Adaptive Convolution (2017, methodology): “In the future, we plan to address this issue by exploring multi-scale strategies, such as those used for optical flow estimation [37].”
Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation (2017, methodology): “Two recent methods have also been proposed [22, 31] to build the classical principles of optical flow into the network architecture,”
Real-Time Intermediate Flow Estimation for Video Frame Interpolation (2020, background): “Thanks related researchers for the exploration of efficient models [48,36,14].”
Sim2real transfer learning for 3D human pose estimation: motion to the rescue (2019, background): “Optical ﬂow is already known to transfer well across domains, because it relies more on similarities between frames than on recognizing speciﬁc patterns [15, 17, 29, 45, 65, 66].”
Learning to See Physics via Visual De-animation (2017, methodology): “It first obtains flow fields from I1 to I2 and from I2 to I3 by a pretrained spatial pyramid network (SPyNet) [Ranjan and Black, 2017].”
Flow-Grounded Spatial-Temporal Video Prediction from Still Images (2018, methodology): “Similar to [14,43], we employ an existing optical flow estimator SPyNet [41] to obtain flows between GT frames for training the 3D-cVAE.”
AutoFlow: Learning a Better Training Set for Optical Flow (2021, methodology): “Spy-Net [35], PWC-Net [45], and LiteFlowNet [16] were designed using several wellestablished principles for optical flow.”
Unifying Flow, Stereo and Depth Estimation (2022, background): “Further advances of network architectures and training strategies [16], [17], [21], [31], [43], [44], [45], [46], [47], [48], [49], [50], [51], [52], [53] have led to steady progress for learning-based methods, which today outperform traditional approaches by a large margin and are currently dominat”
BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment (2021, methodology): “We use pretrained SPyNet [8] as our flow network.”
STM: SpatioTemporal and Motion Encoding for Action Recognition (2019, background): “Recently many approaches have been proposed to estimate optical ﬂow with CNN [5, 14, 6, 21] or explored alternatives of optical ﬂow [33, 39, 26, 18].”
Dynamic Video Segmentation Network (2018, background): “One is called spatial pyramid network (SpyNet) [43], which uses the coarse-tofine spatial pyramid structure of [44] to learn residual flow at each pyramid level.”