Practical and Consistent Estimation of f-Divergences
By Paul K. Rubenstein, Olivier Bousquet, Josip Djolonga, Carlos Riquelme, Ilya O. Tolstikhin. 2019
52 citations recorded by the source corpus.
The estimation of an f-divergence between two probability distributions based on samples is a fundamental problem in statistics and machine learning. Most works study this problem under very weak assumptions, in which case it is provably hard. We consider the case of stronger structural assumptions that are commonly satisfied in modern machine learning, including representation learning and generative modelling with autoencoder architectures. Under these assumptions we propose and study an estimator that can be easily implemented, works well in high dimensions, and enjoys faster rates of convergence. We verify the behavior of our estimator empirically in both synthetic and real-data experiments, and discuss its direct implications for total correlation, entropy, and mutual information estimation.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Probabilistic & Bayesian Methods
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Neil Houlsby
- Xiaohua Zhai
- Lucas Beyer
- Alexander Kolesnikov
- Sylvain Gelly
- Michael Tschannen
- Mario Lucic
- Joan Puigcerver
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Jitendra Malik
- Trevor Darrell
- Neil Houlsby
- Ming-Hsuan Yang
- Xiaogang Wang
- Andrew Zisserman
In Statistical and Nonlinear Physics
- Sylvain Gelly
- Mario Lucic
- Michael Tschannen
- Alexander Kolesnikov
- Marcin Michalski
- Olivier Bousquet
- Olivier Bachem
- Xiaohua Zhai
In Probabilistic & Bayesian Methods
- Mario Lucic
- Sylvain Gelly
- Olivier Bachem
- Michael Tschannen
- Alexander Kolesnikov
- Carlos Riquelme
- Olivier Bousquet
- Xiaohua Zhai
Citations in the corpus
- A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark (2019, methodology): “We evaluate VAEs (Kingma & Welling, 2014), and WAEs with three distribution matching losses: GAN, MMD (Tol-stikhin et al., 2018), and UKL (Rubenstein et al., 2019).”
- The Visual Task Adaptation Benchmark (2019, methodology): “We test VAEs (Kingma & Welling, 2014), and WAEs with three distribution matching losses: GAN, MMD (Tolstikhin et al., 2018), and UKL (Rubenstein et al., 2019).”