Understanding Agent Incentives using Causal Influence Diagrams. Part I: Single Action Settings

33 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Alignment of Language Agents (2021, background): “…stance towards the agent (Dennett, 1989), which means it’s not important whether the agent’s primary objective has an obvious physical correlate, as long as treating the system as an agent optimizing for that primary objective is a good model for predicting its behaviour (Everitt et al., 2019a).”
Defining and Characterizing Reward Hacking (2022, unknown): “, 2011), constrained RL (Szepesvári, 2020), quantilizers (Taylor, 2016), and incentive management (Everitt et al., 2019).”
How RL Agents Behave When Their Actions Are Modified (2021, methodology): “We use the concept of incentives [6, 7] to concisely describe learning behaviours in MAMDPs, complementing the formal analysis of learning algorithm properties.”
Modeling AGI Safety Frameworks with Causal Influence Diagrams (2019, methodology): “This problematic incentive is represented in the diagram in Figure 9, where we can see that the QA-system has both incentive and ability to affect the world state with its answer [Everitt et al. , 2019].”