Entropy and Information Theory

By Robert M. Gray. 1990

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Privacy Aware Learning (2012, background): “Our results can be viewed as rate distortion theorems [14, 8] (with source P and channel Q) for certain compact alphabets, though as far as we know, they are all new.”
Minimax Optimal Procedures for Locally Private Estimation (2016, positive): “Indeed, we have by the standard chain rule for mutual information (Gray 1990, chap.”
Global Intrusion Detection in the DOMINO Overlay System (2004, methodology): “Our approach utilizes the well known Kullback-Leibler [15] distance metric for probability distributions to measure the information gain.”
Information-Based Complexity, Feedback and Dynamics in Convex Programming (2010, background): “The results in [1] are very wide in scope and cover a variety of convex programming problems in Banach spaces; ﬁnite-dimensional versions are covered in [2] and [3].”
PAC-Bayesian Inequalities for Martingales (2011, background): “Martingales are one of the most basic and important tools for studying time-evolving processes and we believe that our results will be useful for multiple domains.”
Introduction to Data Compression (2005, background): “Gray [17]; Elements of Information Theory, by T.”
Learning to Optimize via Information-Directed Sampling (2014, background): “6 of Gray [16], this is equal to the expected reduction in entropy of the posterior distribution of A due to observing Yt(a): gt(a) = E [H(αt)−H(αt+1)|Ft−1, At = a] , (4)”
Calibrating Noise to Variance in Adaptive Data Analysis (2017, unknown): “+Lap(˜σ)) = σ σ˜ e− |µ˜−µ| σ − 1 − |µ˜ −µ| σ + σ ˜σ −1 −ln σ σ˜ ≤ (˜µ−µ)2 2σ˜σ + 1 7 σ˜2 σ2 −1 !2 σ2 σ˜2 . Next we have a technical lemma relating expectations to KL divergence. Lemma 2.6 ([Gra11], Theorem 5.2.1). Let P and Q be probability distributions on Ω. Then D(PkQ) = sup f:Ω→R E X∼P [f(X”
Rethinking and Improving the Robustness of Image Style Transfer (2021, background): “Figure 3 presents the average of maximum values max i,k F li,k and max i,j G li,j , as well as normalized entropies [11] of the activations and Gram matrices, respectively, of the last layer of each network stage of the random models on 10 style images, where i, j, k are the spatial coordinates…”
Computational Mechanics: Pattern and Prediction, Structure and Simplicity (1999, unknown): “[62] for details on how to construct such a mutual information with full rigor.)”
Information Theoretic Clustering (2002, background): “Cross-entropy is also called directed divergence [10], [17] since it is not symmetrical.”