Minimax-Regret Querying on Side Effects for Safe Optimality in Factored Markov Decision Processes

By Shun Zhang, Edmund H. Durfee, Satinder Singh. 2018

31 citations recorded by the source corpus.

As it achieves a goal on behalf of its human user, an autonomous agent's actions may have side effects that change features of its environment in ways that negatively surprise its user. An agent that can be trusted to operate safely should thus only change features the user has explicitly permitted. We formalize this problem, and develop a planning algorithm that avoids potentially negative side effects given what the agent knows about (un)changeable features. Further, we formulate a provably minimax-regret querying strategy for the agent to selectively ask the user about features that it hasn't explicitly been told about. We empirically show how much faster it is than a more exhaustive approach and how much better its queries are than those found by the best known heuristic.

Subfields: Artificial Intelligence, Computer Networks and Communications, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

Shane Legg

In Reinforcement Learning

Citations in the corpus

Scalable agent alignment via reward modeling: a research direction (2018, background): “• Side-effects (Armstrong & Levinstein, 2017; Zhang et al., 2018b; Krakovna et al., 2018): the agent is not incentivized to reduce effects unrelated to its main objectives, even if those are irreversible or difﬁcult to reverse.”
Avoiding Side Effects By Considering Future Tasks (2020, background): “Minimax-regret querying [26] assumes a factored MDP where the agent is allowed to change some of the features and proposes a criterion for querying the supervisor about changing other features in order to allow for intended effects.”