Learning to Estimate Hidden Motions with Global Motion Aggregation

By Shihao Jiang, Dylan Campbell, Hongdong Li, Richard I. Hartley. 2021

421 citations recorded by the source corpus.

Occlusions pose a significant challenge to optical flow algorithms that rely on local evidences. We consider an occluded point to be one that is imaged in the reference frame but not in the next, a slight overloading of the standard definition since it also includes points that move out-of-frame. Estimating the motion of these points is extremely difficult, particularly in the two-frame setting. Previous work relies on CNNs to learn occlusions, without much success, or requires multiple frames to reason about occlusions using temporal smoothness. In this paper, we argue that the occlusion problem can be better solved in the two-frame case by modelling image self-similarities. We introduce a global motion aggregation module, a transformer-based approach to find long-range dependencies between pixels in the first image, and perform global aggregation on the corresponding motion features. We demonstrate that the optical flow estimates in the occluded regions can be significantly improved without damaging the performance in non-occluded regions. This approach obtains new state-of-the-art results on the challenging Sintel dataset, improving the average end-point error by 13.6% on Sintel Final and 13.7% on Sintel Clean. At the time of submission, our method ranks first on these benchmarks among all published and unpublished approaches. Code is available at https://github.com/zacjiang/GMA.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, methodology): “As far as we are aware, this result is state of the art on Sintel.final (GMA [36] produces slightly better numbers us on the somewhat easier Sintel.clean evaluation set using different training data).”
Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers (2021, methodology): “All-pairs intra-frame correlations were subsequently shown to help resolve correspondence ambiguities [34].”
FlowFormer: A Transformer Architecture for Optical Flow (2022, methodology): “Based on RAFT architecture, many works [25,50,26,58,16] were proposed to either reduce the computational costs or improve the flow accuracy.”
GMFlow: Learning Optical Flow via Global Matching (2021, methodology): “To address occluded and out-ofboundary pixels, we incorporate an additional self-attention layer to propagate the high-quality flow prediction from matched pixels to unmatched ones by exploiting the feature self-similarity [17].”
Grounding Image Matching in 3D with MASt3R (2024, background): “Very reminiscent of optical flow approaches [22,40,42,79,80,86], they are usually employing coarse-to-fine schemes to decrease computational complexity.”
Unifying Flow, Stereo and Depth Estimation (2022, methodology): “Further advances of network architectures and training strategies [16], [17], [21], [31], [43], [44], [45], [46], [47], [48], [49], [50], [51], [52], [53] have led to steady progress for learning-based methods, which today outperform traditional approaches by a large margin and are currently dominat”