Tighter Variational Representations of f-Divergences via Restriction to Probability Measures

By Avraham Ruderman, Mark D. Reid, Dario García-García, James Petterson. 2012

69 citations recorded by the source corpus.

We show that the variational representations for f-divergences currently used in the literature can be tightened. This has implications to a number of methods recently proposed based on this representation. As an example application we use our tighter representation to derive a general f-divergence estimator based on two i.i.d. samples and derive the dual program for this estimator that performs well empirically. We also point out a connection between our estimator and MMD.

Subfields: Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

Citations in the corpus

On Mutual Information Maximization for Representation Learning (2019, methodology): “The second estimator is based on the variational form of the KL divergence due to Nguyen, Wainwright, and Jordan (NWJ) (Nguyen et al., 2010) and takes the form (4) For detailed derivations we refer the reader to (Ruderman et al., 2012; Poole et al., 2019).”
Disentangling the independently controllable factors of variation by interacting with the world (2018, positive): “From [Donsker and Varadhan, 1975, Ruderman et al., 2012] we have Applying this equality to the mutual information where θ is the set of weights shared by the factor generator, the policy network and the encoder.”
Mutual Information Neural Estimation (2018, methodology): “A key technical ingredient of MINE are dual representations of the KL-divergence.”
Fairness-Aware Learning with Restriction of Universal Dependency using f-Divergences (2015, methodology): “Our estimation method can be regarded as a generalization of [12, 14, 18].”
MINE: Mutual Information Neural Estimation (2018, methodology): “We refer to the work by Ruderman et al. (2012) for a derivation of both representations (6) and (10) from unifying point of view of Fenchel duality, in the more general context of f -divergences.”
Bridging the Gap Between $f$-GANs and Wasserstein GANs (2019, methodology): “Our approach is most related to regularized variational f -divergence estimators (Nguyen et al., 2010; Ruderman et al., 2012) and linear f -GANs (Liu et al.”