A Minimum Relative Entropy Principle for Learning and Acting

By Pedro A. Ortega, Daniel A. Braun. 2008

126 citations recorded by the source corpus.

This paper proposes a method to construct an adaptive agent that is universal with respect to a given class of experts, where each expert is designed specifically for a particular environment. This adaptive control problem is formalized as the problem of minimizing the relative entropy of the adaptive agent from the expert that is most suitable for the unknown environment. If the agent is a passive observer, then the optimal solution is the well-known Bayesian predictor. However, if the agent is active, then its past actions need to be treated as causal interventions on the I/O stream rather than normal probability conditions. Here it is shown that the solution to this new variational problem is given by a stochastic controller called the Bayesian control rule, which implements adaptive behavior as a mixture of experts. Furthermore, it is shown that under mild assumptions, the Bayesian control rule converges to the control law of the most suitable expert.

Subfields: Artificial Intelligence, Control and Systems Engineering, Management Science and Operations Research, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Management Science and Operations Research

In Probabilistic & Bayesian Methods

Citations in the corpus

Shaking the foundations: delusions in sequence models for interaction and control (2021, background): “Note that this is Thompson Sampling—not as a heuristic, but derived from ﬁrst principles [14, 15].”
A Free Energy Principle for Biological Systems (2012, background): “Minimising variational free energy (maximising Bayesian model evidence) not only provides a principled explanation for perceptual (Bayesian) inference in the brain but can also explain action and behaviour [15,16].”
Action and Perception as Divergence Minimization (2020, methodology): “To keep the notation simple, we remove these variables from z once they become observed and implicitly condition all terms on them, as dictated by causal interventions (Pearl, 1995; Ortega and Braun, 2010).”
Thompson Sampling for Learning Parameterized Markov Decision Processes (2014, background): “…to show the first known structural ( " gap-dependent " ) regret bounds for Thompson sampling in fixed but unknown parameterized MDPs. Prior to this, Ortega and Braun (2010) investigate the consistency performance of posteriorsampling based control rules, again in the fully Bayesian setting where…”
Meta-learning of Sequential Strategies (2019, background): “15 Thompson sampling can be characterized as sampling the actions directly from the posterior predictive [Ortega and Braun, 2010].”
Thermodynamics as a theory of decision-making with information-processing costs (2012, background): “It is straightforward to see that bounded rational controllers of the form (9) share their structure with Bayes’ rule, where we identify the prior p0(x), the likelihood model e βU(x) and the posterior p(x), normalized by the partition function, thus, establishing a close link between inference and c”