Impossibility and Uncertainty Theorems in AI Value Alignment (or why your AGI should not have a utility function)

By Peter Eckersley. 2018

57 citations recorded by the source corpus.

Utility functions or their equivalents (value functions, objective functions, loss functions, reward functions, preference orderings) are a central tool in most current machine learning systems. These mechanisms for defining goals and guiding optimization run into practical and conceptual difficulty when there are independent, multi-dimensional objectives that need to be pursued simultaneously and cannot be reduced to each other. Ethicists have proved several impossibility theorems that stem from this origin; those results appear to show that there is no way of formally specifying what it means for an outcome to be good for a population without violating strong human ethical intuitions (in such cases, the objective function is a social welfare function). We argue that this is a practical problem for any machine learning system (such as medical decision support systems or autonomous weapons) or rigidly rule-based bureaucracy that will make high stakes decisions about human lives: such systems should not use objective functions in the strict mathematical sense. We explore the alternative of using uncertain objectives, represented for instance as partially ordered preferences, or as probability distributions over total orders. We show that previously known impossibility theorems can be transformed into uncertainty theorems in both of those settings, and prove lower bounds on how much uncertainty is implied by the impossibility results. We close by proposing two conjectures about the relationship between uncertainty in objectives and severe unintended consequences from AI systems.

Subfields: Artificial Intelligence, Safety Research, Alignment & RLHF

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Iason Gabriel
2. Dylan Hadfield-Menell
3. Simon Zhuang
4. Thilo Hagendorff
5. Luke Munn
6. Richard Ngo

In Safety Research

1. Iason Gabriel
2. Dylan Hadfield-Menell
3. Simon Zhuang
4. Thilo Hagendorff
5. Luke Munn

In Alignment & RLHF

1. Iason Gabriel
2. Dylan Hadfield-Menell
3. Simon Zhuang
4. Thilo Hagendorff

Citations in the corpus

• Artificial Intelligence, Values, and Alignment (2020, background): “These processes therefore seem unable to settle deeper questions about standing, measurement, and aggregation (Baum, 2017; Eckersley, 2018).”
• Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences (2020, unknown): “…value with a human who is not perfectly rational (Milli et al., 2017) and that there are fundamental impossibility results regarding value alignment (Eckersley, 2018); however, Eckersley (2018) demonstrate that these impossibility results do not hold if the objective is represented as a set of…”
• The Ethics of AI Ethics: An Evaluation of Guidelines (2019, background): “2018) to reflections on specific problems (Eckersley 2018) and comprehensive AI guidelines (The IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems 2019).”
• Consequences of Misaligned AI (2021, background): “Eckersley (2018) (14) argues against the use of rigid objective functions, stating the necessity of a degree of uncertainty at all times in the objective function.”