Toward Idealized Decision Theory

By Nate Soares, Benja Fallenstein. 2015

37 citations recorded by the source corpus.

This paper motivates the study of decision theory as necessary for aligning smarter-than-human artificial systems with human interests. We discuss the shortcomings of two standard formulations of decision theory, and demonstrate that they cannot be used to describe an idealized decision procedure suitable for approximation by artificial systems. We then explore the notions of policy selection and logical counterfactuals, two recent insights into decision theory that point the way toward promising paths for future research.

Subfields: Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Management Science and Operations Research

1. Andrew Critch
2. Jessica Taylor
3. Nate Soares
4. Scott Garrabrant
5. Benja Fallenstein

Citations in the corpus

• Aligning Superintelligence with Human Interests: A Technical Research Agenda (2015, background): “This does not come for free: by default, agents have incentives to preserve their own preferences, even if those conflict with the intentions of the programmers (Omohundro 2008; Soares et al., forthcoming). Special care is needed to specify agents that avoid the default incentives to manipulate and ”
• Quantilizers: A Safer Alternative to Maximizers for Limited Optimization (2016, background): “action, whose maximization results in causal decision theory; see Soares and Fallenstein [7] for discussion of alternative”
• Agent Foundations for Aligning Machine Intelligence with Human Interests: A Technical Research Agenda (2017, background): “One such toy problem is the “shutdown problem,” which involves designing a set of preferences that incentivize an agent to shut down upon the press of a button without also incentivizing the agent to either cause or prevent the pressing of that button (Soares and Fallenstein 2015a).”
• Parametric Bounded Löb's Theorem and Robust Cooperation of Bounded Agents (2016, methodology): “We write Lang(S) for the language of S, Langr(S) for the formulas in Lang(S) with r free variables, and Const(S) for the set of constants in S (e.g. 0, S0, etc.).”
• Logical Induction (2016, background): “Soares and Fallenstein (2015) point out that some deterministic decision-making algorithms reason about counterpossible questions (“what would happen if my deterministic algorithm had the output a vs b vs c?”).”