Bad Universal Priors and Notions of Optimality

By Jan Leike, Jan Leike, Marcus Hutter. 2015

45 citations recorded by the source corpus.

A big open question of algorithmic information theory is the choice of the universal Turing machine (UTM). For Kolmogorov complexity and Solomonoff induction we have invariance theorems: the choice of the UTM changes bounds only by a constant. For the universally intelligent agent AIXI (Hutter, 2005) no invariance theorem is known. Our results are entirely negative: we discuss cases in which unlucky or adversarial choices of the UTM cause AIXI to misbehave drastically. We show that Legg-Hutter intelligence and thus balanced Pareto optimality is entirely subjective, and that every policy is Pareto optimal in the class of all computable environments. This undermines all existing optimality properties for AIXI. While it may still serve as a gold standard for AI, our results imply that AIXI is a relative theory, dependent on the choice of the UTM.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Information Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Information Theory

Citations in the corpus

Scalable agent alignment via reward modeling: a research direction (2018, background): “…with a correctly aligned reward function, the resulting behavior might still be unaligned because the agent may fail to converge to an optimal policy: even provably Bayes-optimal agents may fail to converge to the optimal policy due to a lack of exploration (Orseau, 2013; Leike & Hutter, 2015).”
A Formal Solution to the Grain of Truth Problem (2016, background): “In this case, a Bayes-optimal agent avoids taking these dangerous actions and therefore will not explore enough to wash out the prior’s bias [LH15a].”
Safely Interruptible Agents (2016, background): “Unfortunately, the optimality criterion of AIXI is Bayesian optimality, which is entirely dependent on the subjective prior and posterior [Leike and Hutter, 2015], and AIXI has been shown to not be weakly asymptotically optimal [Orseau, 2013] without additional exploration [Lattimore and Hutter, 201”
Self-Modification of Policy and Utility Function in Rational Agents (2016, background): “Bayes-optimal agents may not explore sufficiently (Leike and Hutter, 2015).”