BA-Net: Dense Bundle Adjustment Network

By Chengzhou Tang, Ping Tan. International Conference on Learning Representations, 2018

338 citations recorded by the source corpus.

This paper introduces a network architecture to solve the structure-from-motion (SfM) problem via feature-metric bundle adjustment (BA), which explicitly enforces multi-view geometry constraints in the form of feature-metric error. The whole pipeline is differentiable so that the network can learn suitable features that make the BA problem more tractable. Furthermore, this work introduces a novel depth parameterization to recover dense per-pixel depth. The network first generates several basis depth maps according to the input image and optimizes the final depth as a linear combination of these basis depth maps via feature-metric BA. The basis depth maps generator is also learned via end-to-end training. The whole system nicely combines domain knowledge (i.e. hard-coded multi-view geometry constraints) and deep learning (i.e. feature learning and basis depth maps learning) to address the challenging dense SfM problem. Experiments on large scale real data prove the success of the proposed method.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/4ba326228728c7ea03a4de6778d0b428b035d5cd

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

VGGT: Visual Geometry Grounded Transformer (2025, positive): “Recent methods [ 3, 67, 71, 74, 75, 78, 81, 83, 89, 108] explored end-to-end differentiable SfM, where VG-GSfM [83] started to outperform traditional algorithms on challenging phototourism scenarios.”
Continuous 3D Perception Model with Persistent State (2025, neutral): “…replacing hand-crafted components with learning-based alternatives (e.g., substituting conventional feature descriptors [6, 47] with learned ones [17, 20, 63, 73]), integrating data-driven priors [13, 77–79, 93, 97, 105] into the systems, or optimizing entire systems end-to-end [76, 78, 83, 94].”
Unifying Flow, Stereo and Depth Estimation (2022, methodology): “…driven by designing task-speciﬁc models to solve each task independently, and thus a large variety of network architectures [16], [17], [18], [19], [20], [21], [22] have been proposed to handle different tasks, ignoring the fact that many multi-view geometry and motion tasks are fundamentally…”
Point-Based Multi-View Stereo Network (2019, background): “Image Feature Pyramid Learning-based image features are demonstrated to be critical to boosting up dense pixel correspondence quality [29, 23].”
Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video (2019, background): “However, although these supervised methods [5, 6, 18–21] show high-quality flow and depth estimation results, it is expensive to acquire ground truth in real-world scenes.”
Differentiable SLAM-net: Learning Particle SLAM for Visual Navigation (2021, methodology): “For example, CodeSLAM[7] and SceneCode[63] learn a compact representation of the scene; CNN-SLAM[57] learns a CNN-based depth predictor as the front-end of a monocular SLAM system; BA-net [56] learns the feature metric representation and a damping factor for bundle adjustment.”
MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion (2024, background): “Given a set of 2D images, structure from motion (SfM) (Sch¨onberger & Frahm, 2016; Teed & Deng, 2018; Tang & Tan, 2018) or visual SLAM (Teed & Deng, 2021; Mur-Artal et al., 2015; Mur-Artal & Tard´os, 2017; Engel et al., 2014; Newcombe et al., 2011) estimate 3D structure of a scene while also…”
PRNet: Self-Supervised Learning for Partial-to-Partial Registration (2019, methodology): “Rather than introducing another hyperparameter, PRNet uses a sub-network to predict the temperature [4] of the Gumbel–Softmax correspondence, which can be cast as a simpliﬁed version of the actor-critic method.”
Back to the Feature: Learning Robust Camera Localization from Pixels to Pose (2021, background): “This learned representation, inspired by past works on handcrafted and learned features for camera tracking [22, 51, 61, 83, 88, 91], is robust to large illumination or viewpoint changes and provides meaningful gradients for successful alignments despite poor initial pose estimates.”
NICE-SLAM: Neural Implicit Scalable Encoding for SLAM (2021, methodology): “BA-Net [50] and DeepFactors [12] simplify the optimization problem by using a set of basis depth maps.”
NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video (2021, methodology): “We use two different training/validation splits following Atlas [30] (top block) and BA-Net [42] (bottom block).”
Point-NeRF: Point-based Neural Radiance Fields (2022, background): “Multi-view 3D reconstruction has been extensively studied and addressed with a number of structure-from-motion [41, 46, 47] and multi-view stereo techniques [10, 14, 24, 42, 54].”
DUSt3R: Geometric 3D Vision Made Easy (2023, positive): “Recent learning-based approaches formulate this problem either as pose and monocular depth regression [93, 172, 185] or pose and stereo matching [123, 131, 136, 142, 183].”
Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations (2019, background): “However, as SRNs are differentiable with respect to camera parameters; future work may alternatively integrate them with learned algorithms for camera pose estimation [72].”
Consistent video depth estimation (2020, methodology): “using the test split provided by Tang and Tan [2019].”
Adaptive VIO: Deep Visual-Inertial Odometry with Online Continual Learning (2024, background): “Therefore, some propose to combine learning techniques with traditional modules [3, 7, 32].”
Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance (2020, background): “Tang and Tan [53] use a deep neural architecture with an integrated differentiable Bundle Adjustment [55] layer to extract a linear basis for the depth of a reference frame, and features from nearby images and to optimize for the depth and the camera parameters in each forward pass.”
RAFT: Recurrent All-Pairs Field Transforms for Optical Flow (2020, methodology): “This has motivated several works to embed optimization problems into network architectures [Amos and Kolter, 2017; Tang and Tan, 2018].”
Cost Volume Pyramid Based Depth Inference for Multi-View Stereo (2019, methodology): “As raw images vary with illumination changes, we adopt learnable features, which has been demonstrated to be crucial step for extracting dense feature correspondences [42, 38].”
DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras (2021, methodology): “These methods are not full SLAM systems, but instead focus on small scale reconstruction on the order of two [8, 50, 53] up to a dozen frames [59, 46, 47].”