Dynamic Safe Interruptibility for Decentralized Multi-Agent Reinforcement Learning
By El Mahdi El Mhamdi, Rachid Guerraoui, Hadrien Hendrikx, Alexandre Maurer. 2017
29 citations recorded by the source corpus.
In reinforcement learning, agents learn by performing actions and observing their outcomes. Sometimes, it is desirable for a human operator to interrupt an agent in order to prevent dangerous situations from happening. Yet, as part of their learning process, agents may link these interruptions, that impact their reward, to specific states and deliberately avoid them. The situation is particularly challenging in a multi-agent context because agents might not only learn from their own past interruptions, but also from those of other agents. Orseau and Armstrong defined safe interruptibility for one learner, but their work does not naturally extend to multi-agent systems. This paper introduces dynamic safe interruptibility, an alternative definition more suited to decentralized learning problems, and studies this notion in two learning frameworks: joint action learners and independent learners. We give realistic sufficient conditions on the learning algorithm to enable dynamic safe interruptibility in the case of joint action learners, yet show that these conditions are not sufficient for independent learners. We show however that if agents can detect interruptions, it is possible to prune the observations to ensure dynamic safe interruptibility even for independent learners.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
Citations in the corpus
- D2: Decentralized Training over Decentralized Data (2018, methodology): “The decentralized approach for multi-task reinforcement learning was studied in Mhamdi et al. [2017], Omidshafiei et al. [2017].”
- Safe Policy Improvement with Baseline Bootstrapping (2017, background): “, 2019), interruptibility (Orseau & Armstrong, 2016; Guerraoui et al., 2017), or as exploration in a hazardous environment (Schulman et al.”
- Communication Compression for Decentralized Training (2018, methodology): “The decentralized structure can applied in solving of multi-task multi-agent reinforcement learning [Omidshafiei et al., 2017, Mhamdi et al., 2017].”
- Dead-ends and Secure Exploration in Reinforcement Learning (2019, background): “…& Fernández, 2015) is an overloaded term, as it may be considered with respect to parametric uncertainty (Thomas, 2015; Petrik et al., 2016; Laroche et al., 2019), internal uncertainty (Altman, 1999; Carrara et al., 2019), or interruptibility (Orseau & Armstrong, 2016; Guerraoui et al., 2017).”
- Decentralization Meets Quantization (2018, methodology): “The decentralized structure can applied in solving of multi-task multi-agent reinforcement learning [Mhamdi et al., 2017, Omidshafiei et al., 2017].”