Information Theory and Statistics: A Tutorial

By Imre Csiszár, Paul C. Shields. 2004

897 citations recorded by the source corpus.

Subfields: Statistics and Probability, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

In Probabilistic & Bayesian Methods

Citations in the corpus

GAIT: A Geometric Approach to Information Theory (2019, background): “Compared to the family of f -divergences (Csiszár and Shields, 2004), this definition computes point-wise ratios between the similarity profiles KP and KQ rather than the probability masses (or more generally, Radon-Nikodym w.”
f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization (2016, background): “A large class of different divergences are the so called f -divergences [5, 21], also known as the Ali-Silvey distances [1].”
CPR: Classifier-Projection Regularization for Continual Learning (2020, background): “…observation that our CPR term can be further interpreted in terms of information projection (I-projection) formulations (Cover & Thomas, 2012; Murphy, 2012; Csiszár & Matus, 2003; Walsh & Regalia, 2010; Amari et al., 2001; Csiszár & Matus, 2003; Csiszár & Shields, 2004) found in information theory.”
Continuous Doubly Constrained Batch Reinforcement Learning (2021, methodology): “In principle, various f -divergences [10] or Integral Probability Metrics [42] could employed in D(·, ·).”
A Distributional Approach to Controlled Text Generation (2020, background): “Note: Csiszár & Shields (2004) assume a finite X here, but generalizations to infinite (countable and/or continuous) X spaces are possible, see (Csiszar, 1975).”
UnitedQA: A Hybrid Approach for Open Domain Question Answering (2021, background): “, 2017), PDR aims at regularizing the posterior difference between the clean and noisy inputs with regard to the family of f -divergences (Csiszár and Shields, 2004).”
Imitation Learning as f-Divergence Minimization (2019, background): “We consider a broad class of such divergences called f-divergences [10, 27].”
From classification to quantification in tweet sentiment analysis (2016, background): “While KLD is less easy to understand to non-initiates than AE or RAE, its advantage is that it is a very wellknown measure, having been the subject of intense study within information theory (Csiszár and Shields 2004) and, although from a more applicative angle, within the language modelling approac”
Variational Inference with Tail-adaptive f-Divergence (2018, background): “To mitigate this issue, alternative metrics have been studied in the literature, a large portion of which are special cases of f -divergence (e.g., Csiszár & Shields, 2004): where f : R + → R is any convex function. is The most notable class of f -divergence that has been exploited in VI α…”
Perturbed Gibbs Samplers for Generating Large-Scale Privacy-Safe Synthetic Health Data (2013, methodology): “For this reason, Gibbs sampler is sometimes viewed as a special case of MH algorithm with no rejection.”
Information-Theoretic Bounded Rationality (2015, background): “In information geometry, this point is known as the information projection of the prior onto the linear subspace [Csiszár and Schields, 2004].”
Practical and Consistent Estimation of f-Divergences (2019, background): “In this work we consider the well-known family of f -divergences [7, 24] that includes amongst others the KL, Jensen-Shannon (JS), 2, and ↵-divergences as well as the Total Variation (TV) and squared Hellinger (H2) distances, the latter two of which play an important role in the statistics literatur”
L_DMI: An Information-theoretic Noise-robust Loss Function (2019, neutral): “3 (Information-monotonicity [6]).”
Unifying Non-Maximum Likelihood Learning Objectives with Minimum KL Contraction (2011, methodology): “Lemma 4 (Csiszàr & Shields [8]) For two distributions p, q ∈ Ωd, with the distribution operator defined in Eq.”
Learning Sparsely Used Overcomplete Dictionaries via Alternating Minimization (2013, background): “, the recent tutorial [8] for an excellent overview).”
On f-Divergences: Integral Representations, Local Behavior, and Inequalities (2018, background): “[16] I. Csiszár and P. C. Shields, “Information Theory and Statistics: A Tutorial”, Foundations and Trends in Communications and Information Theory, vol. 1, no. 4, pp. 417–528, 2004.”
Universal Estimation of Information Measures for Analog Sources (2009, background): “In addition to their prominent role in information theory, they have found numerous applications, among others, in probability theory [13, 19], ergodic theory [218], statistics [64, 142], convex analysis and inequalities [69], physics [25, 27, 147, 150], chemistry [79], molecular biology [270], ecol”