FlowFormer: A Transformer Architecture for Optical Flow

By Zhaoyang Huang, Xiaoyu Shi, Chao Zhang, Qiang Wang, Ka Chun Cheung, Hongwei Qin, Jifeng Dai, Hongsheng Li. European Conference on Computer Vision, 2022

459 citations recorded by the source corpus.

We introduce optical Flow transFormer, dubbed as FlowFormer, a transformer-based neural network architecture for learning optical flow. FlowFormer tokenizes the 4D cost volume built from an image pair, encodes the cost tokens into a cost memory with alternate-group transformer (AGT) layers in a novel latent space, and decodes the cost memory via a recurrent transformer decoder with dynamic positional cost queries. On the Sintel benchmark, FlowFormer achieves 1.159 and 2.088 average end-point-error (AEPE) on the clean and final pass, a 16.5% and 15.5% error reduction from the best published result (1.388 and 2.47). Besides, FlowFormer also achieves strong generalization performance. Without being trained on Sintel, FlowFormer achieves 1.01 AEPE on the clean pass of Sintel training set, outperforming the best published result (1.29) by 21.7%.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/7bffc157b3b3626a3912a3b0ef74ce5904630fce

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

CoTracker: It is Better to Track Together (2023, methodology): “Flowformer [18] drew inspiration from RAFT and proposed a transformer-based approach that tokenizes the 4D cost volume.”
SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentation (2022, background): “Speaking of the recent popular vision transformers [17, 51, 82, 74, 73, 50, 80, 33, 49, 88], they usually ignore adaptability in channel dimension.”
Human Preference Score: Better Aligning Text-to-image Models with Human Preference (2023, background): “Datasets of generated images play a vital role in computer vision tasks that has difficulty in ground-truth acquisition, such as optical flow estimation [24, 7, 4, 44, 18, 39, 40].”
One Fits All: Power General Time Series Analysis by Pretrained LM (2023, methodology): “XGBoost Chen & Guestrin (2016), Rocket Dempster et al. (2020), LSTNet Lai et al. (2018), LSSL Gu et al. (2021), Pyraformer Liu et al. (2021), TCN Franceschi et al. (2019) and Flowformer Huang et al. (2022) are used for classification.”
Grounding Image Matching in 3D with MASt3R (2024, background): “Very reminiscent of optical flow approaches [22,40,42,79,80,86], they are usually employing coarse-to-fine schemes to decrease computational complexity.”