Information and Entropy

By A. Caticha. 2007

58 citations recorded by the source corpus.

What is information? Is it physical? We argue that in a Bayesian theory the notion of information must be defined in terms of its effects on the beliefs of rational agents. Information is whatever constrains rational beliefs and therefore it is the force that induces us to change our minds. This problem of updating from a prior to a posterior probability distribution is tackled through an eliminative induction process that singles out the logarithmic relative entropy as the unique tool for inference. The resulting method of Maximum relative Entropy (ME), which is designed for updating from arbitrary priors given information in the form of arbitrary constraints, includes as special cases both MaxEnt (which allows arbitrary constraints) and Bayes' rule (which allows arbitrary priors). Thus, ME unifies the two themes of these workshops—the Maximum Entropy and the Bayesian methods—into a single general inference scheme that allows us to handle problems that lie beyond the reach of either of the two methods separately. I conclude with a couple of simple illustrative examples.

Subfields: Cognitive Neuroscience, History and Philosophy of Science, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

1. Rafael Chaves
2. Dominik Janzing
3. B. Schoelkopf
4. Hans J. Briegel
5. Bernhard Schölkopf

In History and Philosophy of Science

1. Rafael Chaves
2. Dominik Janzing
3. B. Schoelkopf
4. Bernhard Schölkopf

In Statistical and Nonlinear Physics

1. Rafael Chaves
2. B. Schoelkopf
3. Dominik Janzing
4. Hans J. Briegel

Citations in the corpus

• Algorithmic independence of initial condition and dynamical law in thermodynamics and causal inference (2015, methodology): “For imperfect knowledge of the microscopic state, Zurek [18] considers thermodynamic entropy as the sum of statistical entropy and Kolmogorov complexity [25], which thus unifies the statistical and the algorithmic perspective of physical entropy.”
• QUANTUM KOLMOGOROV COMPLEXITY AND ITS APPLICATIONS (2006, background): “First of all we recall how the Kolmogorov complexity became important in the context of thermodynamics and review some results presented by Caves [10, 11] in this context.”