Avoiding Side Effects By Considering Future Tasks
By Victoria Krakovna, Laurent Orseau, Richard Ngo, Miljan Martic, Shane Legg. 2020
51 citations recorded by the source corpus.
Designing reward functions is difficult: the designer has to specify what to do (what it means to complete the task) as well as what not to do (side effects that should be avoided while completing the task). To alleviate the burden on the reward designer, we propose an algorithm to automatically generate an auxiliary reward function that penalizes side effects. This auxiliary objective rewards the ability to complete possible future tasks, which decreases if the agent causes side effects during the current task. The future task reward can also give the agent an incentive to interfere with events in the environment that make future tasks less achievable, such as irreversible actions by other agents. To avoid this interference incentive, we introduce a baseline policy that represents a default course of action (such as doing nothing), and use it to filter out future tasks that are not achievable by default. We formally define interference incentives and show that the future task approach with a baseline policy avoids these incentives in the deterministic case. Using gridworld environments that test for side effects and interference, we show that our method avoids interference and is more effective for avoiding side effects than the common approach of penalizing irreversible actions.
Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jacob Steinhardt
- Dan Hendrycks
- Nicholas Carlini
- Percy Liang
- Sergey Levine
- Dawn Song
- Quoc V. Le
- John Schulman
In Computational Theory and Mathematics
- Iason Gabriel
- Tom Everitt
- Geoffrey Irving
- Nicholas Carlini
- Dan Hendrycks
- Zachary Kenton
- John Schulman
- Jacob Steinhardt
In Reinforcement Learning
- Iason Gabriel
- Tom Everitt
- Jacob Steinhardt
- Vladimir Mikulik
- Geoffrey Irving
- John Schulman
- Laura Weidinger
- Zachary Kenton
In Safety, Robustness & Interpretability
- Jacob Steinhardt
- Dan Hendrycks
- Nicholas Carlini
- John Schulman
- Iason Gabriel
- Matthias Hein
- Tom Everitt
- Dawn Song
Citations in the corpus
- Alignment of Language Agents (2021, methodology): “As discussed by Krakovna et al. (2020b) we are still at the early stages of finding solution approaches for objective gaming.”
- Unsolved Problems in ML Safety (2021, background): “Likewise, researchers could create agents that properly account for their lack of knowledge of the true objective [69] and avoid disrupting parts of the environment whose value is unclear [190, 103, 159].”