Unifying Flow, Stereo and Depth Estimation

By Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, Fisher Yu, Dacheng Tao, Andreas Geiger. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022

395 citations recorded by the source corpus.

We present a unified formulation and model for three motion and 3D perception tasks: optical flow, rectified stereo matching and unrectified stereo depth estimation from posed images. Unlike previous specialized architectures for each specific task, we formulate all three tasks as a unified dense correspondence matching problem, which can be solved with a single model by directly comparing feature similarities. Such a formulation calls for discriminative feature representations, which we achieve using a Transformer, in particular the cross-attention mechanism. We demonstrate that cross-attention enables integration of knowledge from another image via cross-view interactions, which greatly improves the quality of the extracted features. Our unified model naturally enables cross-task transfer since the model architecture and parameters are shared across tasks. We outperform RAFT with our unified model on the challenging Sintel dataset, and our final model that uses a few additional task-specific refinement steps outperforms or compares favorably to recent state-of-the-art methods on 10 popular flow, stereo and depth datasets, while being simpler and more efficient in terms of model design and inference speed.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/01c630e9ce06040b0f3f540dd3f7681f01e84a9d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation (2024, methodology): “In contrast, OpenVid - 1M adopts a more efficient UniMatch [28] to achieve better optical flows, addressing not only static videos but also those with fast movements.”
MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images (2024, methodology): “…far depth ranges, we first uniformly sample D depth candidates { d m } Dm =1 in the inverse depth domain and then warp view j ’s feature F j to view i with the camera projection matrices P i , P j and each depth candidate d m , to obtain D warped features where W denotes the warping operation [46].”
Metric3D v2: A Versatile Monocular Geometric Foundation Model for Zero-Shot Metric Depth and Surface Normal Estimation (2024, neutral): “Currently, the community still lacks a robust, gener-alizable geometry foundation model [14], [15], [16] capable of producing high-quality metric depth and surface normal from a single image.”