Approximating the Kullback Leibler Divergence Between Gaussian Mixture Models

By John R. Hershey, Peder A. Olsen. 2007

1,303 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Learning Sparse Neural Networks through L0 Regularization (2017, methodology): “…Since both q ( z ) and p ( z ) are mixtures with the same number of components we can use the chain rule of relative entropy (Cover & Thomas, 2012; Hershey & Olsen, 2007) in order to compute the KL-divergence: KL ( q ( z ) || p ( z )) = Q ¯ s (0) log Q ¯ s (0) P ¯ s (0) + (1 − Q ¯ s (1)) log 1 −…”
Learning dexterous manipulation for a soft robotic hand from human demonstrations (2016, methodology): “Methods such as MCMC sampling can be used to estimate it, but we find a variational upper bound [26] to be the most suitable for our formulation.”
On entropy approximation for Gaussian mixture random vectors (2008, background): “A lower bound of (3) can be obtained by employing an upper bound of the Kullback-Leibler divergence (see [17]) between two Gaussian mixtures, which is derived in [7].”
Anomaly Detection and Localization in Crowded Scenes (2014, positive): “We adopt the variational approximation of [34]: KLðp XjC kp X Þ % X i C i log P K C j C j exp À ÀKL À p i XjC p j XjC ÁÁ P K0þK1 j ! j exp À ÀKL À p i XjC p j X ÁÁ ( ) : ð8Þ Each term of (8) contains a KL divergence between DTs, which can be computed in closed form [35].”
Distance based similarity models for content based multimedia retrieval (2013, methodology): “This Monte Carlo sampling is one method to estimate the KullbackLeibler Divergence with arbitrary accuracy [Hershey and Olsen, 2007].”
Super-human multi-talker speech recognition: A graphical modeling approach (2010, methodology): “Under this model the d Gaussians are optimized by minimizing the KL-divergence D s ð P s a pðs a Þpðx a js a Þjj P s a pðs a Þ^ pðx a js a ÞÞ, and likewise for s b , using the variational approximation of Hershey and Olsen (2007).”
Deep Private-Feature Extraction (2018, methodology): “[46] For two multidimensional Gaussian distributions, p and q, with mp and mq as their expected values and the same covariance matrix sI, we have:”
Bootstrapping Single-channel Source Separation via Unsupervised Spatial Clustering on Stereo Mixtures (2018, background): “(5) has no closed-form solution for arbitrary mixture models [19, 20].”
Markov-switching dynamic factor models in real time (2018, background): “After some straightforward algebra in the last two expressions, it is easy to show that if st+1 = 1, then using yk;t+1 is helpful to reduce false signals when yk;t+1 < k ( 0 + 1) 2 (15)”
Practical and Consistent Estimation of f-Divergences (2019, background): “Chi square divergence between two d -variate Gaussian distributions both having covariance matrices proportional to identity can be computed as: The squared Hellinger divergence between two Gaussians is given in [33]: The KL -divergence between two d -variate Gaussians is:”
Anomaly detection in crowded scenes (2010, methodology): “In this work, we adopt the variational approximation proposed in [14]”
Expandable Data-Driven Graphical Modeling of Human Actions Based on Salient Postures (2008, methodology): “We adopt the variational estimation of KL divergence recently proposed by Hershey and Olsen [48].”
Comparison of approximation methods to Kullback–Leibler divergence between Gaussian mixture models for satellite image retrieval (2016, methodology): “Although it is simple to formulate, but the identiﬁcation property (Hershey and Olsen 2007) does not hold and it is prone to underestimate D ( X || Y ).”