SfM-Net: Learning of Structure and Motion from Video

By Sudheendra Vijayanarasimhan, Susanna Ricco, Cordelia Schmid, Rahul Sukthankar, Katerina Fragkiadaki. arXiv.org, 2017

465 citations recorded by the source corpus.

We propose SfM-Net, a geometry-aware neural network for motion estimation in videos that decomposes frame-to-frame pixel motion in terms of scene and object depth, camera motion and 3D object rotations and translations. Given a sequence of frames, SfM-Net predicts depth, segmentation, camera and rigid object motions, converts those into a dense frame-to-frame motion field (optical flow), differentiably warps frames in time to match pixels and back-propagates. The model can be trained with various degrees of supervision: 1) self-supervised by the re-projection photometric error (completely unsupervised), 2) supervised by ego-motion (camera motion), or 3) supervised by depth (e.g., as provided by RGBD sensors). SfM-Net extracts meaningful depth estimates and successfully estimates frame-to-frame camera rotations and translations. It often successfully segments the moving objects in the scene, even though such supervision is never provided.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/ab6b8671d3baef80f87ad76a3e4ce452aa1b6467

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

SuperPoint: Self-Supervised Interest Point Detection and Description (2017, background): “Joint pose and depth estimation models also exist [33, 30, 28], but do not use interest points.”
Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction (2018, background): “But in fact it is possible to go further and to use deep networks also to estimate the camera ego-motion, as shown very recently by [44] and [37], both of which use a photometric error for supervising a monocular depth and ego-motion estimation system.”
Unsupervised Video Object Segmentation for Deep Reinforcement Learning (2018, methodology): “This loss is chosen instead of the L1 loss used to train SfM-Net [31].”
Learning to Find Good Correspondences (2017, methodology): “In contrast with traditional methods, many of these new techniques [32, 31, 37, 36] are dense, using the raw image as input.”
Monocular depth estimation based on deep learning: An overview (2020, methodology): “Since the flow field of rigid regions in [85], [15] is generated by the depth and pose estimation,”
TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement (2023, background): “They are mainly used in structure-from-motion (SFM) settings and typically ignore occlusion, as SFM can use geometry to filter errors [19, 47, 60, 61].”
Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation (2018, background): “More recent works [20, 28, 29, 32, 33] have approached estimation of these problems by coupling two or more problems together in an unsupervised learning framework.”
Digging Into Self-Supervised Monocular Depth Estimation (2018, methodology): “Like other works [76, 61, 38], we also apply a per-pixel mask μ to the loss, selectively weighting pixels.”
TAP-Vid: A Benchmark for Tracking Any Point in a Video (2022, unknown): “Structure-From-Motion (SFM) [12, 30, 43, 48] relies on sparse keypoint matches given image pairs of rigid scenes, and typically ignores occlusion, as it can use geometry to filter errors [21, 55, 71, 73].”
Instance Embedding Transfer to Unsupervised Video Object Segmentation (2018, methodology): “SfMNet [39] uses differentiable rendering to learn object masks and motion models without mask annotations.”
LF-Net: Learning Local Features from Images (2018, background): “This remains true even in competition with deep network alternatives, which typically involve dense matching [9, 43, 45, 52, 53] and tend to work best on narrow baselines, as they can suffer from occlusions, which local features are robust against.”
MODNet: Motion and Appearance based Moving Object Detection Network for Autonomous Driving (2018, background): “A recent trend [29][12][4][31][6] for learning motion in videos has emerged.”
Neural Expectation Maximization (2017, methodology): “More recently optical ﬂow has been used to train convolutional networks to do ﬁgure/ground segmentation [23, 34].”
Stereo magnification (2018, background): “Finally, a variety of work in computer vision has used view synthesis as an indirect form of supervision for other tasks, such as predicting depth, shape, or optical flow from one or more images [Garg and Reid 2016; Godard et al. 2017; Liu et al. 2017; Tulsiani et al. 2017; Vijayanarasimhan et al. 2”
Rethinking the Faster R-CNN Architecture for Temporal Action Localization (2018, methodology): “To compute optical flow, we use a FlowNet [11] model trained on artificially generated data followed by fine-tuning on the Kinetics dataset using an unsupervised loss [44].”
Self-Supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidance (2020, methodology): “Our method is also related to approaches that rely on state-of-the-art segmentation techniques [5,6,39,55,57], which either give these as an additional input to the network [39] or use it to predict the relative pose between the same DC object in two consecutive frames [5,6,57], using this informati”
DF-Net: Unsupervised Joint Learning of Depth and Flow using Cross-Task Consistency (2018, methodology): “The synthesized flow from depth and pose can either be used for flow prediction in rigid regions [60,65,68,48] as is or used for view synthesis to train depth model using monocular videos [73].”
DeepV2D: Video to Depth with Differentiable Structure from Motion (2018, methodology): “This is different from prior deep-learning based approaches [49, 42, 41, 48], which incorporate geometry to a substantially lesser extent by re-1 quiring the network to learn some known geometric operations, e.g. motion estimation from correspondence.”
GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose (2018, background): “[50] learned several object masks and corresponding rigid motion parameters for modelling moving objects.”