A Bayesian Rule for Adaptive Control based on Causal Interventions
By Pedro A. Ortega, Daniel A. Braun. 2009
23 citations recorded by the source corpus.
Explaining adaptive behavior is a central problem in artificial intelligence research. Here we formalize adaptive agents as mixture distributions over sequences of inputs and outputs (I/O). Each distribution of the mixture constitutes a ‘possible world’, but the agent does not know which of the possible worlds it is actually facing. The problem is to adapt the I/O stream in a way that is compatible with the true world. A natural measure of adaptation can be obtained by the KullbackLeibler (KL) divergence between the I/O distribution of the true world and the I/O distribution expected by the agent that is uncertain about possible worlds. In the case of pure input streams, the Bayesian mixture provides a well-known solution for this problem. We show, however, that in the case of I/O streams this solution breaks down, because outputs are issued by the agent itself and require a different probabilistic syntax as provided by intervention calculus. Based on this calculus, we obtain a Bayesian control rule that allows modeling adaptive behavior with mixture distributions over I/O streams. This rule might allow for a novel approach to adaptive control based on a minimum KLprinciple.
Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Pieter Abbeel
- Yoshua Bengio
- Ruslan Salakhutdinov
- Chelsea Finn
- Quoc V. Le
- J. Urgen Schmidhuber
- Trevor Darrell
In Computational Theory and Mathematics
- Joel Veness
- Daniel A. Braun
- Sergey Levine
- Shane Legg
- Nando de Freitas
- Pedro A. Ortega
- Marcus Hutter
- Scott E. Reed
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Nando de Freitas
- Bilal Piot
- Shane Legg
- Marc G. Bellemare
- R´emi Munos
- Pedro A. Ortega
Citations in the corpus
- Shaking the foundations: delusions in sequence models for interaction and control (2021, background): “In addition, the resulting policy is adaptive—essentially, it is Thompson sampling—and will converge to the expert’s policy under broad conditions [12, 14, 16].”
- A Minimum Relative Entropy Principle for Learning and Acting (2008, methodology): “In this subsection, this procedure will be used to derive an adaptive agent (Ortega and Braun, 2010a).”