Two problems with variational expectation maximisation for time-series models

By Richard E. Turner, Maneesh Sahani. 2011

222 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Statistics and Probability, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

In Probabilistic & Bayesian Methods

Citations in the corpus

Meta-Learning Probabilistic Inference for Prediction (2018, background): “This is due to VI’s tendency to under-fit, especially for small numbers of data points (Trippe and Turner, 2018; Turner and Sahani, 2011) which is compounded when using inference networks (Cremer et al.”
Generalized Variational Continual Learning (2020, methodology): “When applied to multi-head networks, VCL tends to prune out large parts of the network (Trippe & Turner, 2018; Turner & Sahani, 2011) and GVCL inherits this behaviour.”
Neural Autoregressive Flows (2018, background): “This is important since a poor variational approximation to the posterior can fail to reﬂect the right amount of uncertainty , and/or be biased (Turner & Sahani, 2011), resulting in inaccurate and un-reliable predictions.”
Fixed-Form Variational Posterior Approximation through Stochastic Linear Regression (2012, background): “This obviously works well in capturing the dependencies of the model, but it also generally works well for capturing the posterior dependencies induced by the likelihood.”
Scalable Variational Inference for Bayesian Variable Selection in Regression, and Its Accuracy in Genetic Association Studies (2012, background): “The tendency to concentrate more mass on a single hypothesis, or to artificially lower the variance in the posterior by overly favouring the winner, is typical behaviour of mean field approximations (MacKay 2003; Turner et al. 2008).”
Boosting Variational Inference (2016, background): “But this assumption also means that MFVI eﬀectively cannot capture multimodality and that MFVI tends to underestimate the posterior covariance, sometimes drastically [Bishop, 2006, Wang and Titterington, 2004, Turner and Sahani, 2011, Rue et al., 2009, MacKay, 2003].”
AUTO-ENCODING VARIATIONAL BAYES (2020, unknown): “The variational distribution recovered by AEVB, which minimizes the reverse Kullback-Leibler (KL) divergence between the variational distribution and the model posterior, is known to systematically underestimate variance [11, 12], making it arbitrarily bad as an importance sampling proposal distribu”
Gradient Estimators for Implicit Models (2017, background): “M L ] 19 M a y 2017 often introduce undesirable biases, responsible for problems such as mode collapse in the context of GANs, and the underestimation of uncertainty in variational inference methods [44].”
A Structured Model of Video Reproduces Primary Visual Cortical Organisation (2009, background): “In practice, image models have often been fit by maximising the data probability under settings of both the parameters and the unobserved variables (in our case these would be the identity and attribute variables), a procedure which may be severely suboptimal [40].”
Disentangling Disentanglement in Variational Autoencoders (2018, background): “(2019), increasing β can reinforce existing inductive biases, wherein mean-field assumptions encourage representations which reduce dependence between the latent dimensions (Turner and Sahani, 2011).”