Penalizing Side Effects using Stepwise Relative Reachability

By Victoria Krakovna, Laurent Orseau, Miljan Martic, Shane Legg. 2018

64 citations recorded by the source corpus.

How can we design safe reinforcement learning agents that avoid unnecessary disruptions to their environment? We show that current approaches to penalizing side effects can introduce bad incentives, e.g. to prevent any irreversible changes in the environment, including the actions of other agents. To isolate the source of such undesirable incentives, we break down side effects penalties into two components: a baseline state and a measure of deviation from this baseline state. We argue that some of these incentives arise from the choice of baseline, and others arise from the choice of deviation measure. We introduce a new variant of the stepwise inaction baseline and a new deviation measure based on relative reachability of states. The combination of these design choices avoids the given undesirable incentives, while simpler baselines and the unreachability measure fail. We demonstrate this empirically by comparing different combinations of baseline and deviation measure choices on a set of gridworld experiments designed to illustrate possible bad incentives.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Scalable agent alignment via reward modeling: a research direction (2018, background): “• Side-effects (Armstrong & Levinstein, 2017; Zhang et al., 2018b; Krakovna et al., 2018): the agent is not incentivized to reduce effects unrelated to its main objectives, even if those are irreversible or difﬁcult to reverse.”
Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective (2019, background): “Whether it is feasible to robustly specify such a reward function for more complex and open-ended environments is an important open question that is perhaps best answered empirically (Krakovna et al., 2019; Leike, Krueger, et al., 2018).”
Defining and Characterizing Reward Hacking (2022, unknown): “Optimizing a learned proxy reward can be dangerous, however; for instance, it might overlook side-effects (Krakovna et al., 2018; Turner et al., 2019) or encourage power-seeking (Turner et al., 2021) behavior.”
Consequences of Misaligned AI (2021, background): “Initial research in impact minimization (3; 6; 23) attempts to do this by restricting changes to the overall state of the world, but this will likely remain a challenging idea to implement robustly.”
Understanding Agent Incentives using Causal Influence Diagrams. Part I: Single Action Settings (2019, background): “, 2017), side effects (Armstrong and Levinstein, 2017; Krakovna et al., 2019), and boxing/containment (Babcock et al.”
Avoiding Side Effects By Considering Future Tasks (2020, methodology): “The future task approach is similar to relative reachability [10] and attainable utility [24].”