Continuous-Time Flows for Efficient Inference and Density Estimation

By Changyou Chen, Chunyuan Li, Liquan Chen, Wenlin Wang, Yunchen Pu, Lawrence Carin. 2017

62 citations recorded by the source corpus.

Two fundamental problems in unsupervised learning are efficient inference for latent-variable models and robust density estimation based on large amounts of unlabeled data. Algorithms for the two tasks, such as normalizing flows and generative adversarial networks (GANs), are often developed independently. In this paper, we propose the concept of {\em continuous-time flows} (CTFs), a family of diffusion-based methods that are able to asymptotically approach a target distribution. Distinct from normalizing flows and GANs, CTFs can be adopted to achieve the above two goals in one framework, with theoretical guarantees. Our framework includes distilling knowledge from a CTF for efficient inference, and learning an explicit energy-based distribution with CTFs for density estimation. Both tasks rely on a new technique for distribution matching within amortized learning. Experiments on various tasks demonstrate promising performance of the proposed CTF framework, compared to related techniques.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Chunyuan Li
2. Lawrence Carin
3. Zhe Gan
4. Dinghan Shen
5. Jun Zhu
6. Frank No'e
7. Wenlin Wang
8. Chongxuan Li

In Computer Vision and Pattern Recognition

1. Chunyuan Li
2. Lawrence Carin
3. Zhe Gan
4. Jun Zhu
5. Luc Van Gool
6. Dacheng Tao
7. Chongxuan Li
8. Jianfeng Gao

In Statistical and Nonlinear Physics

1. Frank No'e
2. Hao Wu
3. Jonas Köhler
4. Lars Ruthotto
5. Leon Klein
6. Chongxuan Li
7. Cheng Lu
8. Samy Wu Fung

In Generative Models

1. Frank No'e
2. Jonas Köhler
3. Chunyuan Li
4. Jun Zhu
5. Lawrence Carin
6. Leon Klein
7. Chongxuan Li
8. Zhe Gan

In Probabilistic & Bayesian Methods

1. Frank No'e
2. Jonas Köhler
3. Leon Klein
4. Hao Wu
5. Chunyuan Li
6. Cheng Lu
7. Samy Wu Fung
8. Zhe Gan

Citations in the corpus

• Stochastic Normalizing Flows (2020, methodology): “(Chen et al., 2017) propose a learnable stochastic process by integrating Langevin dynamics with learnable drift and diffusion term.”
• Equivariant Flows: exact likelihood generative learning for symmetric densities (2020, methodology): “We will provide a possible implementations based on the recently introduced framework of continuous normalizing flows (CNFs) (Chen et al., 2017).”
• OT-Flow: Fast and Accurate Continuous Normalizing Flows via Optimal Transport (2020, methodology): “They perform well in density estimation (Chen et al. 2018a; Grathwohl et al. 2019; Papamakarios et al. 2019) and inference (Ingraham et al. 2019; Papamakarios et al. 2019), especially in physics and computational chemistry (Noé et al. 2019; Brehmer et al. 2020).”
• Towards Better Representations with Deep/Bayesian Learning (2018, background): “…VAEs and GANs (Pu et al., 2015; Stevens et al., 2017; Fan et al., 2016a; Pu et al., 2016a,b; Chen et al., 2018b; Li et al., 2018a; Pu et al., 2017b; Chen et al., 2018a; Xu et al., 2017; Wang et al., 2017b; Pu et al., 2018b; Yuan et al., 2017, 2018; Chapfuwa et al., 2018), sigmoid belief networks…”
• Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limit (2019, methodology): “Since the marginal distributions of such a process evolve through a deterministic continuous-time flow, one can use ODE techniques in this context as well [Tabak and Vanden-Eijnden, 2010, Chen et al., 2018a].”
• Implicit Normalizing Flows (2021, background): “Continuous Time Flows (CTFs) (Chen et al., 2018b; Grathwohl et al., 2019; Chen et al., 2018a) are flexible alternative to discrete time flows for generative modeling.”
• Topic Compositional Neural Language Model (2017, background): “Variational inference has been successfully applied in a variety of applications (Pu et al., 2016; Wang et al., 2017; Chen et al., 2017).”
• Monge-Ampère Flow for Generative Modeling (2018, background): “(2015); Chen et al. (2017); Frogner & Poggio (2018) consider generative models based on diffusion processes.”