Information Theory and Statistical Mechanics

By E. Jaynes. 1957

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Theory

Citations in the corpus

Direct-coupling analysis of residue coevolution captures native contacts across many protein families (2011, methodology): “" i , A i : Besides this constraint, we aim at the most general, least constrained model This can be achieved by applying the maximum-entropy principle (45, 46), and leads to an explicit mathematical form of !”
Reliable Uncertainty Estimates in Deep Neural Networks using Noise Contrastive Priors (2018, background): “Output regularization is a classic idea from the maximum entropy principle (Jaynes, 1957), where it has motivated label smoothing (Szegedy et al.”
Variational Bayesian Reinforcement Learning with Regret Bounds (2018, background): “(8) describes the probability that the system at temperature τt is in a particular ‘state’ [22].”
Sinkhorn Distances: Lightspeed Computation of Optimal Transport (2013, methodology): “Note that this is equivalent to considering the maximum-entropy principle (Jaynes, 1957; Darroch and Ratcliff, 1972) if we were to maximize entropy while keeping the transportation cost constrained.”
Protein structure prediction from sequence variation (2012, methodology): “One global statistical approach is known as entropy maximization under data constraints, a classic inference method connecting information theory and Boltzmann statistics 35 .”
An Algorithmic Perspective on Imitation Learning (2018, methodology): “The maximum entropy principle in [Jaynes, 1957] suggests to choose a distribution that maximizes the entropy among the distributions that achieve at least the same total reward.”
Incremental Learning with Maximum Entropy Regularization: Rethinking Forgetting and Intransigence (2019, methodology): “The maximum entropy principle or regularization (MER) is proposed to prevent overﬁtting by enforcing the exploration in predicted labels (Jaynes, 1957).”
The Fitness Landscape of HIV-1 Gag: Advanced Modeling Approaches and Validation of Model Predictions by In Vitro Testing (2014, methodology): “Mathematically, ‘‘least biased’’ implies the distribution that has maximum entropy in the information-theoretic sense [22].”
Fast and Accurate Multivariate Gaussian Modeling of Protein Families: Predicting Residue Contacts and Protein-Interaction Partners (2014, neutral): “DCA is based on the maximum-entropy (MaxEnt) principle [19,20] which naturally leads to statistical models of protein families in terms of so-called Potts models or Markov random fields.”
Coordinated Multi-Agent Imitation Learning (2017, methodology): “To resolve this confusion, we employ the well-known principle of maximum entropy (Jaynes, 1957), which is equivalent to regularizing the imitation loss function with some low entropy penalty (Grandvalet et al., all roles are occupied across all execution sequences.”
The Near-Earth Space Radiation Environment (2008, methodology): “Jaynes showed in his studies of statistical mechanics that the distribution that should be selected is the one that maximizes the mathematical expression for entropy subject to the constraints imposed by available information.”
Regret in the Newsvendor Model with Partial Information (2008, background): “The distribution that maximizes the entropy is thus a good prior distribution because it is the “maximally noncommittal with regard to missing information” (Jaynes 1957).”
Inverse statistical problems: from the inverse Ising problem to data science (2017, background): “We will discuss in section II A 3 how this distribution emerges as the ‘least biased distribution’ of a binary random variable with prescribed first and second moments [107].”
Human Neuroscience (unknown): “…). q ⋅ We make minimal assumptions about the form of the approximate posteriors by following the maximum entropy principle: given knowledge of, or assumptions about, constraints on a distribution, the least arbitrary choice of distribution is the one that maximizes entropy (Jaynes, 1957).”