On Measures of Entropy and Information

By Gavin E. Crooks. 2015

Subfields: Probabilistic & Bayesian Methods, Information Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

In Information Theory

Citations in the corpus

GAIT: A Geometric Approach to Information Theory (2019, background): “It is evident that whenever K = I, this definition reduces to the Rényi entropy (Rényi, 1961).”
Measuring Data (2022, background): “Diversity is also referred to as heterogeneity in many sciences (Rényi et al., 1961; Nunes et al., 2020) which have their own approaches to measure it, Hill numbers in ecology (Hill, 1973), Hannah–Kay indices in economics (Hannah and Kay, 1977).”
Privacy Amplification by Iteration (2018, neutral): “Definition 4 (Rényi Divergence [29]).”
Algorithms and Theory for Multiple-Source Adaptation (2018, methodology): “We followed the standard protocol from Saenko et al. (2010), whereby 20 labeled examples are available for training from the amazon domain and 8 labeled examples are available from both the webcam and dslr domains.”
Rényi Differential Privacy (2017, background): “The (parameterized) Rényi divergence is classically defined as follows [18]:”
Testing Hypotheses by Regularized Maximum Mean Discrepancy (2013, methodology): “Erdogmus and Principe [5] showed that − log ‖μP‖(2)H is the Parzen window estimation of the Renyi entropy [16].”
Learning to select data for transfer learning with Bayesian Optimization (2017, neutral): “• Rényi divergence (Rényi, 1961): ) .”
Past, Present, and Future of Simultaneous Localization and Mapping: Toward the Robust-Perception Age (2016, neutral): “While the former requires recognizing a previously seen place (disregarding whether that place is a kitchen, a corridor, etc.), the latter is interested in classifying the place according to semantic labels.”
A New Weibull-G Family of Distributions (2015, methodology): “Here, we derive explicit expressions for the Rényi entropy [25] of the NWG family.”
Description of quantum coherence in thermodynamic processes requires constraints beyond free energy (2014, background): “Here γ is the thermal state of the system with Hamiltonian H, γ = e/ZH , β = (kT )−1 and Sα (sometimes denoted Dα) are information-theoretic generalizations of the standard relative entropy, called α-Rényi divergences [29] (See methods).”
Multiple-source adaptation theory and algorithms (2020, result): “The one that naturally comes up in our analysis and in previous work [30, 40, 41] is the R´enyi Diver-gence [48].”
A review of source term estimation methods for atmospheric dispersion events using static or mobile sensors (2017, methodology): “forms, such as Kullback-Lieber divergence [155] (variation of entropy), Rényi divergence [156] or a measure of the mutual information.”
Some parsimonious models for lifetimes and applications (2021, background): “Reńyi entropy (Reńyi [17]) is one of the well known entropy measures.”
Algorithmic Transparency via Quantitative Input Influence: Theory and Experiments with Learning Systems (2016, background): “Quantitative Information Flow traces its information-theoretic roots to the work of Shannon [42] and Rényi [43].”
The Zografos–Balakrishnan-G Family of Distributions: Mathematical Properties and Applications (2015, background): “No general properties are derived in Ristić and Balakrishnan (2012). The aim of this article is to derive some mathematical properties of (1) and (2) in the most simple, explicit, and general forms.”
Estimating Renyi Entropy of Discrete Distributions (2014, background): “Since Rényi entropy is a measure of randomness (see [36] for a detailed discussion), it is maximized by the uniform distribution and the following inequalities hold: 0 ≤ Hα(p) ≤ log k, α = 1,”
Rapid Convergence of the Unadjusted Langevin Algorithm: Isoperimetry Suffices (2019, background): “Rényi divergence is a family of generalizations of KL divergence [56, 59, 11], which becomes stronger as the order q increases.”