Information, Divergence and Risk for Binary Experiments
By Mark D. Reid, Robert C. Williamson. 2009
250 citations recorded by the source corpus.
We unify f-divergences, Bregman divergences, surrogate regret bounds, proper scoring rules, cost curves, ROC-curves and statistical information. We do this by systematically studying integral and variational representations of these objects and in so doing identify their representation primitives which all are related to cost-sensitive binary classification. As well as developing relationships between generative and discriminative views of learning, the new machinery leads to tight and more general surrogate regret bounds and generalised Pinsker inequalities relating f-divergences to variational divergence. The new viewpoint also illuminates existing algorithms: it provides a new derivation of Support Vector Machines in terms of divergences and relates maximum mean discrepancy to Fisher linear discriminants.
Subfields: Artificial Intelligence, Statistical Learning Theory, Information Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Bernhard Schölkopf
- Yoshua Bengio
- Dacheng Tao
- Sergey Levine
- Arthur Gretton
- Alex Smola
- Ruslan Salakhutdinov
- Quoc V. Le
In Statistical Learning Theory
- Arthur Gretton
- Bernhard Schölkopf
- Kenji Fukumizu
- Bharath K. Sriperumbudur
- Alex Smola
- Karsten M. Borgwardt
- Le Song
- Balaji Lakshminarayanan
In Information Theory
- Balaji Lakshminarayanan
- Shivani Agarwal
- Igal Sason
- Shakir Mohamed
- Marco Loog
- Clayton D. Scott
- Gavin E. Crooks
- Bernhard Schölkopf
Citations in the corpus
- f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization (2016, neutral): “An equivalent result has also been derived by Reid and Williamson [28].”
- Learning in Implicit Generative Models (2016, methodology): “…we described previously, by cleverly formulating our problem as a Bregman divergence minimisation, we show that all the methods we described for density ratio estimation are very closely related (Reid and Williamson, 2011; Sriperumbudur et al., 2009; Sugiyama et al., 2012a;b; Uehara et al., 2016).”
- Stabilizing Training of Generative Adversarial Networks through Regularization (2017, background): “An interesting property of f -divergences is that they permit a variational characterization Nguyen et al. (2010); Reid & Williamson (2011) via where d P /d Q is the Radon-Nikodym derivative and f c ( t ) ≡ sup u ∈ dom f { ut − f ( u ) } is the convex conjugate of f .”
- Revisiting Classifier Two-Sample Tests (2016, neutral): “), discussed in ( ??”
- Generalization Bounds for Domain Adaptation (2012, background): “The quantity DF(S, T ) is termed as the integral probability metric that has played an important role in probability theory for measuring the difference between the two probability distributions (see Zolotarev, 1984; Rachev, 1991; Müller, 1997; Reid and Williamson, 2011).”
- AdaGAN: Boosting Generative Models (2017, methodology): “For further details on f -divergences we refer to Section 1.3 of [10] and [14].”
- $f$ -Divergence Inequalities (2015, background): “(37) Remark 1: Different functions may lead to the same f -divergence for all (P,Q): if for an arbitrary b ∈ R, we have fb(t) = f0(t) + b (t− 1), t ≥ 0 (38) then Df0(P‖Q) = Dfb(P‖Q).”
- Divergence for s-concave and log concave functions (2013, background): “ of f-divergences whose definition is given in Section 2. Such divergences and their related inequalities are important tools in information theory, statistics, probability theory and machine learning [8, 13, 18, 24, 32, 33, 53, 55, 71]. Consequently, it is desirable to have such divergences availabl”
- Of quantiles and expectiles: consistent scoring functions, Choquet representations and forecast rankings (2015, background): “…investigated independently for quite some time (see for example Bartlett et al. (2006), Tewari and Bartlett (2005), Zhang (2004), Steinwart (2007), Reid and Williamson (2011), Calauzènes et al. (2012, 2013), Scott (2012), Duchi et al. (2013), Menon and Williamson (2014), Agarwal and Agarwal…”
- Contrastive Pessimistic Likelihood Estimation for Semi-Supervised Classification (2015, background): “It is a glimpse of the earlier mentioned difficult interrelationship two different performance criteria can display [73], [76], [77], [78], which we alluded to earlier on in Section 5.”
- Calibrated asymmetric surrogate losses (2012, background): “This bound is equivalent to the bound presented in Theorem 3.1 below, which is obtained by generalizing the approach of Bartlett et al. For completeness, this alternate perspective is presented in Appendix A. Reid and Williamson [23, 25] also study α-classification calibrated losses and derive surro”
- On f-Divergences: Integral Representations, Local Behavior, and Inequalities (2018, background): “, [4], [42], [43], [54], [69], [72], [73]).”
- Clustering: Science or Art? (2009, background): “Formally the choice of a notion of information is tantamount to the choice of a loss function in a supervised learning problem (Reid and Williamson, 2009).”
- Robust Solutions of Optimization Problems Affected by Uncertain Probabilities (2013, unknown): “We choose φ -divergences because these play a fundamental role in statistics (see [30] and [35]).”
- Under Review as a Conference Paper at Iclr 2017 Learning in Implicit Generative Models (2017, methodology): “…by cleverly formulating our problem as a Bregman divergence minimisation, we show that all the methods we described for density ratio estimation are very closely related (Sugiyama et al., 2012a; Reid and Williamson, 2011; Sriperumbudur et al., 2009; Sugiyama et al., 2012b; Uehara et al., 2016).”
- A Kernel Two-Sample Test (2012, background): “Reid and Williamson (2011) provide further discussion and examples.”