Composing graphical models with neural networks for structured representations and fast inference

By Matthew J. Johnson, David Kristjanson Duvenaud, Alexander B. Wiltschko, Ryan P. Adams, Sandeep R. Datta. 2016

520 citations recorded by the source corpus.

We propose a general modeling and inference framework that composes probabilistic graphical models with deep learning methods and combines their respective strengths. Our model family augments graphical structure in latent variables with neural network observation models. For inference, we extend variational autoencoders to use graphical model approximating distributions with recognition networks that output conjugate potentials. All components of these models are learned simultaneously with a single objective, giving a scalable algorithm that leverages stochastic variational inference, natural gradients, graphical model message passing, and the reparameterization trick. We illustrate this framework with several example models and an application to mouse behavioral phenotyping.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Probabilistic & Bayesian Methods, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Probabilistic & Bayesian Methods

In Signal Processing

Citations in the corpus

Recurrent Neural Filters: Learning Independent Bayesian Filtering Steps for Time Series Prediction (2019, methodology): “Hybrid Approaches: In [23], the authors take a hybrid approach with the structured variational autoencoder (SVAE), proposing an efficient general inference framework that combines probabilistic graphical models for the latent state with neural network observation models.”
Probabilistic Model-Agnostic Meta-Learning (2018, methodology): “We can further avoid storing a separate variational distribution qi(φi|θ) and qi(θ) for each task Ti by employing an amortized variational inference technique [22, 21, 36], where we instead set qi(φi|θ) = qψ(φi|θ,x tr i ,y tr i ,x test i ,y test i ), where qψ is defined by some function approximator”
OptNet: Differentiable Optimization as a Layer in Neural Networks (2017, neutral): “Other work (Johnson et al., 2016; Amos et al., 2016) (the later with some inequality constraints) uses argmin differentiation within the context of a specific optimization problem, but doesn’t consider a particularly general setting; similarly, the older work of (Mairal et al.”
A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning (2017, background): “A different combination of VAEs and probabilistic graphical models has been explored in [15], which defines a general class of models where inference is performed with message passing algorithms that use deep neural networks to map the observations to conjugate graphical model potentials.”
Learning Disentangled Representations with Semi-Supervised Deep Generative Models (2017, background): “145 147 q � z ( z y , x ) q � y ( y x ) , decomposes 148 We will consider models in which both ) and q φ ( y , z | x ) can have arbitrary conditional dependency structures | | q � y ( y | x ) alongside the generative condi-149 tioned on the selected class.”
Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data (2017, background): “While there has been much research investigating static data, such as the aforementioned ones, there is relatively little research on learning from sequential data [8, 3, 2, 9, 7, 18, 36].”
Multi-Level Variational Autoencoder: Learning Disentangled Representations from Grouped Observations (2017, methodology): “While any distribution could be employed, we focus on using a product of Normal density functions, which can be seen as an instance of Structured Variational Autoencoders (SVAE) (Johnson et al. 2016).”
Recent Advances in Autoencoder-Based Representation Learning (2018, methodology): “To combine the tractability of conjugate graphical model inference with the ﬂexibility of VAEs, Johnson et al. [24] employ inference models that output conjugate graphical model potentials [58] instead of the parameters of the approximate posterior distribution.”
Automatic differentiation in machine learning: a survey (2015, methodology): “cent interest is implicit and iterative AD techniques (Griewank and Walther, 2008), which has found use in work incorporating constrained optimization within deep learning (Amos and Kolter, 2017) and probabilistic graphical models and neural networks (Johnson et al., 2016).”
Discrete Variational Autoencoders (2016, methodology): “Structured variational autoencoders use conjugate priors to construct powerful approximating posterior distributions (Johnson et al., 2016).”
Harnessing Deep Neural Networks with Logic Rules (2016, background): “…to recent neural autoencoders (Kingma and Welling, 2014; Rezende et al., 2014) where generative models encode probabilistic structures and neural recognition models distill the information through iterative optimization (Rezende et al., 2016; Johnson et al., 2016; Karaletsos et al., 2016).”
Disentangled Sequential Autoencoder (2018, methodology): “Structured (Johnson et al., 2016) and Fac-torised VAEs (Deng et al., 2017) used hierarchical priors to learn more interpretable latent variables.”
Structured Disentangled Representations (2018, background): “me or all of the data [Kingma et al., 2014; Siddharth et al., 2017]. In this latter work, the overall training objective is largely kept the same, and instead the focus is on deﬁning structured VAEs [Johnson et al., 2016] that incorporating a graphical model structure which partitions the latent var”
Disentanglement by Nonlinear ICA with General Incompressible-flow Networks (GIN) (2020, methodology): “Several unsupervised VAE models implement conditioning in the latent space by means of Gaussian mixtures (Johnson et al., 2016; Dilokthanakul et al., 2016; Zhao et al., 2019).”