Sliced Wasserstein Distance for Learning Gaussian Mixture Models

By Soheil Kolouri, Gustavo Kunde Rohde, Heiko Hoffmann. 2017

143 citations recorded by the source corpus.

Gaussian mixture models (GMM) are powerful parametric tools with many applications in machine learning and computer vision. Expectation maximization (EM) is the most popular algorithm for estimating the GMM parameters. However, EM guarantees only convergence to a stationary point of the log-likelihood function, which could be arbitrarily worse than the optimal solution. Inspired by the relationship between the negative log-likelihood function and the Kullback-Leibler (KL) divergence, we propose an alternative formulation for estimating the GMM parameters using the sliced Wasserstein distance, which gives rise to a new algorithm. Specifically, we propose minimizing the sliced-Wasserstein distance between the mixture model and the data distribution with respect to the GMM parameters. In contrast to the KL-divergence, the energy landscape for the sliced-Wasserstein distance is more well-behaved and therefore more suitable for a stochastic gradient descent scheme to obtain the optimal GMM parameters. We show that our formulation results in parameter estimates that are more robust to random initializations and demonstrate that it can estimate high-dimensional data distributions more faithfully than the EM algorithm.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

Citations in the corpus

Complementary Learning for Overcoming Catastrophic Forgetting Using Experience Replay (2019, background): “For these reasons, common distribution distance measures such as KL divergence and Jensen–Shannon divergence are not suitable [Kolouri et al., 2018].”
Sliced Wasserstein Discrepancy for Unsupervised Domain Adaptation (2019, background): “The Wasserstein distance has also recently raised interest in stabilizing generative modeling [1, 14, 73], learning introspective neural networks [32], and obtaining Gaussian mixture models [29] thanks to its geometrically meaningful distance measure even when the supports of the distributions do no”
Generalized Sliced Wasserstein Distances (2019, background): “Therefore, it has recently attracted ample attention and successfully been applied to a variety of practical tasks [22, 24, 25, 26, 27, 28, 29, 30, 31].”
Deep Transfer Learning for Few-Shot SAR Image Classification (2019, methodology): “In this paper, we used the Sliced Wasserstein Distance (SWD) [19], which is a good approximate of optimal transport [20] and additionally can be computed more efﬁciently.”
Wasserstein Distributionally Robust Optimization: Theory and Applications in Machine Learning (2019, methodology): “In the wider context of machine learning, for instance, the Wasserstein distance is used for the analysis of mixture models [56, 70] as well as for image processing [1, 33, 55, 75, 105], computer vision and graphics [78, 79, 90, 102, 103], data-driven bioengineering [34, 57, 108], clustering [49], d”
SAR Image Classification Using Few-Shot Cross-Domain Transfer Learning (2019, methodology): “In this paper, we use the Sliced Wasserstein Distance (SWD) [27] is a good approximate of optimal transport [17] and can be computed more efficiently.”
Learning Transferable Knowledge Through Embedding Spaces (2019, methodology): “As a result, it has non-vanishing gradients, and firstorder gradient-based optimization algorithms can be used to solve optimization problems involving SWD terms [157; 104].”