Prediction-Based Multi-Agent Reinforcement Learning in Inherently Non-Stationary Environments

By Andrei Marinescu, Ivana Dusparic, Siobhán Clarke. 2017

72 citations recorded by the source corpus.

Multi-agent reinforcement learning (MARL) is a widely researched technique for decentralised control in complex large-scale autonomous systems. Such systems often operate in environments that are continuously evolving and where agents’ actions are non-deterministic, so called inherently non-stationary environments. When there are inconsistent results for agents acting on such an environment, learning and adapting is challenging. In this article, we propose P-MARL, an approach that integrates prediction and pattern change detection abilities into MARL and thus minimises the effect of non-stationarity in the environment. The environment is modelled as a time-series, with future estimates provided using prediction techniques. Learning is based on the predicted environment behaviour, with agents employing this knowledge to improve their performance in realtime. We illustrate P-MARL’s performance in a real-world smart grid scenario, where the environment is heavily influenced by non-stationary power demand patterns from residential consumers. We evaluate P-MARL in three different situations, where agents’ action decisions are independent, simultaneous, and sequential. Results show that all methods outperform traditional MARL, with sequential P-MARL achieving best results.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Matthew Riemer
2. Irina Rish
3. Khimya Khetarpal
4. Doina Precup
5. Changxi Zhu
6. Yi Cai
7. Ho-fung Leung
8. Shuyue Hu

In Electrical and Electronic Engineering

1. Renzhi Lu
2. Seung Ho Hong
3. Yi Cai
4. Doina Precup
5. Irina Rish
6. Matthew Riemer
7. Mengmeng Yu

Citations in the corpus

• Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “However, this framework has also has been readily applicable to more challenging multi-agent learning settings (Da Silva et al., 2006; Amato et al., 2013; Marinescu et al., 2017; Vezhnevets et al., 2019).”
• Incentive-based demand response for smart grid with reinforcement learning and deep neural network (2019, background): “In [32,33], the authors demonstrated a prediction-based multi-agent RL approach to minimize the effect of non-stationarity in smart grid scenarios, in which the prediction and pattern change detection abilities were integrated into multi-agent RL.”
• A Q-values Sharing Framework for Multi-agent Reinforcement Learning under Budget Constraint (2020, background): “(MAS), which is also referred as Mutiagent Reinforcement Learning (MARL), these problems are further intensified since all agents are changing and adapting their behaviors [3], thereby increasing the amount of interactions needed.”