Agent Incentives: A Causal Perspective
By Tom Everitt, Ryan Carey, Eric Langlois, Pedro A. Ortega, Shane Legg. 2021
71 citations recorded by the source corpus.
We present a framework for analysing agent incentives using causal influence diagrams. We establish that a well-known criterion for value of information is complete. We propose a new graphical criterion for value of control, establishing its soundness and completeness. We also introduce two new concepts for incentive analysis: response incentives indicate which changes in the environment affect an optimal decision, while instrumental control incentives establish whether an agent can influence its utility via a variable X. For both new concepts, we provide sound and complete graphical criteria. We show by example how these results can help with evaluating the safety and fairness of an AI system
Subfields: Artificial Intelligence, Causal Inference & Discovery
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Iason Gabriel
- Tom Everitt
- Geoffrey Irving
- Vladimir Mikulik
- Laura Weidinger
- Zachary Kenton
- Richard Ngo
- Samuel R. Bowman
In Causal Inference & Discovery
Citations in the corpus
- Alignment of Language Agents (2021, methodology): “Secondly, even if the supplied objective is correct, a capable agent may still exhibit undesired behaviour due to secondary objectives that arise in pursuit of its primary objective, such as tampering with its feedback channel (Everitt et al., 2021b).”