Estimating divergence functionals and the likelihood ratio by penalized convex risk minimization

By XuanLong Nguyen, Martin J. Wainwright, Michael I. Jordan. 2007

154 citations recorded by the source corpus.

Subfields: Statistics and Probability, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

In Statistical Learning Theory

Citations in the corpus

Kernel Mean Embedding of Distributions: A Review and Beyonds (2016, methodology): “…using the kernel density estimate (e.g., L2 divergence, Rényi, Kullback-Leibler, or many other divergences) avoids the need to estimate the density by adopting a direct divergence approximation using, e.g., density ratio estimation and regression (Nguyen et al. 2007, Póczos et al. 2011).”
GenDICE: Generalized Offline Estimation of Stationary Values (2020, background): “Classical density ratio estimation includes moment matching (Gretton et al., 2008), probabilistic classification (Bickel et al., 2007), and ratio matching (Nguyen et al., 2008; Sugiyama et al., 2008; Kanamori et al., 2009).”
A Kernel Method for the Two-Sample-Problem (2006, methodology): “ace. Harchaoui et al. (2008) build on the work of Gretton et al. (2007a), considering a homogeneity statistic arising from the kernel Fisher discriminant, rather than the diﬀerence of RKHS means; and Nguyen et al. (2008) obtain a KL divergence estimate by approximat30 A Kernel Method forthe Two-Samp”
Estimating Divergence Functionals and the Likelihood Ratio by Convex Risk Minimization (2008, methodology): “As a particular case of the Kullback-Leibler divergence, the mutual information specifies capacities in (1)Preliminary versions of this work were presented at the International Symposium on Information Theory (2007) [18] and the Neural Information Processing Systems Conference (2007) [17].”
Scalable Kernel Methods via Doubly Stochastic Gradients (2014, background): “i= 2 i( 1(z i)exp(f(y i))˚ ! i (y i) 0(z i)˚ ! i (x i)); z i˘Bernoulli(0:5): In particular, the x i and y i are not sampled in pair, they are sampled independently from P(x) and Q(x) respectively. [24] proposed another convex optimization based on r KL(˝) whose solution is a nonparametric estimator”
Improved generator objectives for GANs (2016, background): “The key result they leverage from [8] is that any f -divergence can be lower-bounded by where f (cid:63) is the Fenchel conjugate 2 of f , and T is the variational function also known as the discriminator in the GAN literature 3 .”
A Kernel Two-Sample Test (2012, unknown): “Harchaoui et al. (2008) define a two-sample test statistic arising from the kernel Fisher discriminant, rather than the difference of RKHS means; and Nguyen et al. (2008) obtain a KLdivergence estimate by approximating the ratio of densities (or its log) with a function in an RKHS.”
Wild Variational Approximations (2016, methodology): “Neither the variational techniques for estimating f -divergence [43, 44] as the exact posterior is difficult to sample from.”
Estimation of Information Theoretic Measures for Continuous Random Variables (2008, methodology): “Also, in [20, 21], the authors compute the divergence using a variational approach, in which”
Kernel Measures of Independence for non-iid Data (2008, background): “Unfortunately, it typically involves density estimation or at least a nontrivial optimization procedure [11].”
Recovering Best Statistical Guarantees via the Empirical Divergence-Based Distributionally Robust Optimization (2016, methodology): “librating the ball radius requires nonparametricdivergence estimation, which involves challenging statistical analyses on bandwidth tuning and loss of estimation eﬃciency (e.g., Moon and Hero (2014), Nguyen et al. (2007), Pa´l et al. (2010)). The empirical DRO based on the EL framework cleanly bypas”
Stochastic Neighbor Embedding under f-divergences (2018, background): “We explored minimizing the ft-SNE criteria by expressing it in terms of its conjugate dual [17, 16]: Df (P ||Q) = ∑ i 6=j [ qij ( sup h∈H h((xi, xj)) pij qij − f(h((xi, xj))) )]”
Conditional Density Estimation via Least-Squares Density Ratio Estimation (2010, methodology): “Other methods of direct density ratio estimation (Sugiyama et al., 2008; Nguyen et al., 2008) employs the Kullback-Leibler (KL) divergence as the loss function, instead of the squared-loss.”