Nonparametric Divergence Estimation with Applications to Machine Learning on Distributions

By Barnab'as P'oczos, Liang Xiong, Jeff Schneider. 2011

111 citations recorded by the source corpus.

Low-dimensional embedding, manifold learning, clustering, classification, and anomaly detection are among the most important problems in machine learning. The existing methods usually consider the case when each instance has a fixed, finite-dimensional feature representation. Here we consider a different setting. We assume that each instance corresponds to a continuous probability distribution. These distributions are unknown, but we are given some i.i.d. samples from each distribution. Our goal is to estimate the distances between these distributions and use these distances to perform low-dimensional embedding, clustering/classification, or anomaly detection for the distributions. We present estimation algorithms, describe how to apply them for machine learning tasks on distributions, and show empirical results on synthetic data, real word images, and astronomical data sets.

Subfields: Artificial Intelligence, Probabilistic & Bayesian Methods, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Probabilistic & Bayesian Methods

In Signal Processing

Citations in the corpus

Kernel Mean Embedding of Distributions: A Review and Beyonds (2016, methodology): “Secondly, compared to density estimation approaches (Póczos et al. 2011, Oliva et al. 2014; 2013), the kernel mean representation is less prone to the curse of dimensionality and admits fast convergence in terms of sample size (Shawe-Taylor and Cristianini 2004, Smola et al. 2007, Grünewälder et…”
Learning Theory for Distribution Regression (2014, background): “One might also plug into learning algorithms (based on similarities of distributions) consistent Rényi and Tsallis divergence estimates (Póczos et al., 2011, 2012), but these similarity indices are not kernels, and their consistency in specific learning tasks, similarly to the previous works, is…”
One-Class Support Measure Machines for Group Anomaly Detection (2013, methodology): “In this experiment, we are interested in identifying anomalous groups of galaxies, as previously studied in Póczos et al. (2011) and Xiong et al. (2011a;b).”
Point Cloud GAN (2018, background): “In many statistically guaranteed distance estimates, the adopted statistics are commonly based on distance between nearest neighbors (Wang et al., 2009; Póczos et al., 2012).”
Distribution to Distribution Regression (2013, methodology): “Futhermore, kernels based nonparametric estimators of divergences have also been explored (Póczos et al., 2012a;b).”
Support Measure Data Description A One-Class Classifier for Group Anomaly Detection (2017, methodology): “We used the same seing described in [19, 20, 31] to construct the non-anomalous groups.”
FairTest: Discovering Unwarranted Associations in Data-Driven Applications (2015, background): “MI can be defined for continuous O and S but is more expensive to estimate, especially in high dimensions [39], [40].”
Discovering Unwarranted Associations in Data-Driven Applications with the FairTest Testing Toolkit (2015, background): “, with MI [44]) is hard, so it is common to consider specialized relationships for such variables.”
On Estimating L 22 Divergence (2014, background): “We also show that this estimator is minimax optimal.”
Support Measure Data Description for group anomaly detection (2015, methodology): “However, there is a growing interest in machine learning methods for datasets whose individual observations are clusters, groups or sets of points in R [32, 33, 20, 19, 25, 35, 10, 34, 26, 15, 18].”
Support Distribution Machines (2012, methodology): “Using the same tools as in P´oczos et al. (2011), one can prove that by introducing a multiplicative term the following estimator is L 2 consistent under certain conditions (see the Appendix for details): where .”
Domain Adaptation by Joint Distribution Invariant Projections (2020, methodology): “In particular, we exploit the L2-distance [32], [40] to measure the similarity between joint distributions.”