Moser Flow: Divergence-based Generative Modeling on Manifolds

By N. Rozen, Aditya Grover, Maximilian Nickel, Yaron Lipman. 2021

82 citations recorded by the source corpus.

We are interested in learning generative models for complex geometries described via manifolds, such as spheres, tori, and other implicit surfaces. Current extensions of existing (Euclidean) generative models are restricted to specific geometries and typically suffer from high computational costs. We introduce Moser Flow (MF), a new class of generative models within the family of continuous normalizing flows (CNF). MF also produces a CNF via a solution to the change-of-variable formula, however differently from other CNF methods, its model (learned) density is parameterized as the source (prior) density minus the divergence of a neural network (NN). The divergence is a local, linear differential operator, easy to approximate and calculate on manifolds. Therefore, unlike other CNFs, MF does not require invoking or backpropagating through an ODE solver during training. Furthermore, representing the model density explicitly as the divergence of a NN rather than as a solution of an ODE facilitates learning high fidelity densities. Theoretically, we prove that MF constitutes a universal density approximator under suitable assumptions. Empirically, we demonstrate for the first time the use of flow models for sampling from general curved surfaces and achieve significant improvements in density estimation, sample quality, and training complexity over existing CNFs on challenging synthetic geometries and real-world benchmarks from the earth and climate sciences.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Flow Matching for Generative Modeling (2022, background): “Indeed, maximum likelihood training ( e.g. , Grathwohl et al. (2018)) require expensive numerical ODE simulations, while existing simulation-free methods either involve intractable integrals (Rozen et al., 2021) or biased gradients (Ben-Hamu et al., 2022).”
Improving and generalizing flow-based generative models with minibatch optimal transport (2023, unknown): “Prior work considering simulation-free training of CNFs considers algorithms that are equivalent to CFM with Gaussian source distribution (Rozen et al., 2021; Ben-Hamu et al., 2022; Lipman et al., 2023) or independent samples from q 0 , q 1 (Albergo & Vanden-Eijnden, 2023; Albergo et al., 2023;…”
Building Normalizing Flows with Stochastic Interpolants (2022, unknown): “Moreover, (Rozen et al., 2021) relies on interpolating directly the probability measures, which can lead to unstable velocities.”
Riemannian Score-Based Generative Modeling (2022, methodology): “Note that the loss function used in (Rozen et al., 2021) also involves computing a divergence.”
Matching Normalizing Flows and Probability Paths on Manifolds (2022, methodology): “In the sphere case Sd ⊂ Rd+1, similarly to (Rozen et al., 2021) we define v to be constant in the normal direction to the sphere and produce tangent vectors via the tangent projection operator vθ(t, x) = ( I − xx T ‖x‖2 ) wθ ( t, x ‖x‖ ) , (28) where wθ : Rd+2 → Rd+1 is an MLP.”