Occupancy Flow: 4D Reconstruction by Learning Particle Dynamics

By Michael Niemeyer, Lars M. Mescheder, Michael Oechsle, Andreas Geiger. IEEE International Conference on Computer Vision, 2019

317 citations recorded by the source corpus.

Deep learning based 3D reconstruction techniques have recently achieved impressive results. However, while state-of-the-art methods are able to output complex 3D geometry, it is not clear how to extend these results to time-varying topologies. Approaches treating each time step individually lack continuity and exhibit slow inference, while traditional 4D reconstruction methods often utilize a template model or discretize the 4D space at fixed resolution. In this work, we present Occupancy Flow, a novel spatio-temporal representation of time-varying 3D geometry with implicit correspondences. Towards this goal, we learn a temporally and spatially continuous vector field which assigns a motion vector to every point in space and time. In order to perform dense 4D reconstruction from images or sparse point clouds, we combine our method with a continuous 3D representation. Implicitly, our model yields correspondences over time, thus enabling fast inference while providing a sound physical description of the temporal dynamics. We show that our method can be used for interpolation and reconstruction tasks, and demonstrate the accuracy of the learned correspondences. We believe that Occupancy Flow is a promising new 4D representation which will be useful for a variety of spatio-temporal reconstruction tasks.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/1ea4575839ed2dc80ea8d0ce7ec862f13eb549ba

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Unsupervised Learning of 3D Object Categories from Videos in the Wild (2021, methodology): “Similar to previous methods [37, 40, 35], we exploit the representational power of deep neural networks and define Ψ as a deep multi-layer perceptron (MLP): (c, σ) = Ψnr(x, r, z).”
Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation (2021, background): “While this approach is in principle applicable to non-rigid objects, this remains to be tested, and extensions based on recent work on non-rigid scenes in 3D reconstruction and novel view synthesis [9, 18, 26, 28, 31, 43] might be necessary.”
StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation (2021, background): “Neural implicit representations (NIR) have been widely adopted for learning shape and appearance of objects [4,10, 14, 20, 45, 50, 52, 56, 57], local parts [17, 18], and full 3D scenes [7, 11, 29, 55] from explicit 3D supervisions.”
Implicit Neural Representations with Periodic Activation Functions (2020, background): “Temporally aware extensions [17] and variants that add part-level semantic segmentation [18] have also been proposed.”
Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction (2021, background): “While a large variety of technical approaches was proposed [56, 62, 13, 25, 26, 46, 8], they are predominantly trained and benchmarked either on synthetic data [6], or on real datasets of speciﬁc object categories such as birds [62] or chairs [37].”
Convolutional Occupancy Networks (2020, methodology): “Implicit models have been adopted for learning implicit representations from images [23, 24, 29, 41], for encoding texture information [30], for 4D reconstruction [28] as well as for primitive-based reconstruction [10, 11, 15, 32].”
D-NeRF: Neural Radiance Fields for Dynamic Scenes (2020, background): “Occupancy flow [28] was the first work to tackle non-rigid geometry by learning continuous vector field assigning a motion vector to every point in space and time, but it requires full 3D ground-truth supervision.”
Nerfies: Deformable Neural Radiance Fields (2020, methodology): “OccFlow [39] uses a flow-field to represent 3D human motion over time using an ODE, while our approach does not rely on temporal information.”
I M Avatar: Implicit Morphable Head Avatars from Videos (2021, background): “This is an additional neural network that first maps observations in deformed space to observations in a canonical space, where the implicit function is then evaluated [32, 34, 34, 38, 42, 51, 53].”
DIST: Rendering Deep Implicit Signed Distance Function With Differentiable Sphere Tracing (2019, methodology): “Similar idea has also been used to encode other information such as textures [33, 40] and 4D dynamics [32].”
MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction (2022, background): “While early works [9, 40, 47] relied on 3D supervision, several recent works [44, 62, 76] use differentiable surface rendering to reconstruct scenes from multi-view images.”
pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis (2020, background): “Temporally aware extensions [43] and multimodal variants with partlevel semantic segmentation [28] have also been proposed.”
Volume Rendering of Neural Implicit Surfaces (2021, background): “Recent studies have been focusing on model implicit functions with multi-layer perceptron (MLP) due to its expressive representation power and low memory foot-print, including scene (geometry & appearance) representation [9, 20, 19, 23, 25, 29, 36, 28, 35] and free-view rendering [33, 16, 30, 26, 17”
GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields (2020, background): “Implicit Functions: Using implicit functions to represent 3D geometry has gained popularity in learning-based 3D reconstruction [11, 12, 22, 59, 60, 65, 67, 69, 76] and has been extended to scene-level reconstruction [8, 13, 35, 72, 79].”