Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learning

By Georgios Papoudakis, Filippos Christianos, Arrasy Rahman, Stefano V. Albrecht. 2019

244 citations recorded by the source corpus.

Recent developments in deep reinforcement learning are concerned with creating decision-making agents which can perform well in various complex domains. A particular approach which has received increasing attention is multi-agent reinforcement learning, in which multiple agents learn concurrently to coordinate their actions. In such multi-agent environments, additional learning problems arise due to the continually changing decision-making policies of agents. This paper surveys recent works that address the non-stationarity problem in multi-agent deep reinforcement learning. The surveyed methods range from modifications in the training procedure, such as centralized training, to learning representations of the opponent's policy, meta-learning, communication, and decentralized learning. The survey concludes with a list of open problems and possible lines of future research.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

Citations in the corpus

A survey and critique of multiagent deep reinforcement learning (2018, background): “, search [14], planning [28], and more recently with multiagent systems, with an emerging area of multiagent deep reinforcement learning (MDRL)[29, 30].”
A survey of multi-agent deep reinforcement learning with communication (2022, unknown): “Due to the recent success of deep learning [11] and its application to reinforcement learning [12], multi-agent deep reinforcement learning (MADRL) has witnessed great achievements in recent years, where agents can process high-dimensional data and have generalization ability in large state and…”
Shared Experience Actor-Critic for Multi-Agent Reinforcement Learning (2020, background): “This is a difﬁcult problem due to non-stationarity caused by concurrently learning agents and the fact that the joint action space grows exponentially in the number of agents [26].”
Model-augmented Prioritized Experience Replay (2022, background): “Shared weights are frequently used in multi-agent RL frameworks (Rashid et al., 2018; Papoudakis et al., 2019), mostly for learning the policy networks.”
Influencing Long-Term Behavior in Multiagent Reinforcement Learning (2022, unknown): “The joint learning of agents induces non-stationary environment dynamics from the perspective of each agent, requiring an agent to adapt its behavior with respect to potentially unknown changes in the policies of other agents [2].”
A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learning (2020, background): “This dependence on non-stationary policies renders the Markov property invalid from the perspective of each agent, requiring agents to adapt their behaviors with respect to potentially large, unpredictable, and endless changes in the policies of fellow agents (Papoudakis et al., 2019).”
Revisiting Parameter Sharing in Multi-Agent Deep Reinforcement Learning (2020, methodology): “…of information can greatly slow convergence during learning, especially for highly coordinated tasks with many agents, and this speciﬁc form of nonstationarity is believed to be a fundamental reason why converging to good policies in multi-agent learning is so difﬁcult Papoudakis et al. (2019).”
Towards Open Ad Hoc Teamwork Using Graph-based Policy Learning (2020, methodology): “Multi-agent reinforcement learning (MARL): MARL algorithms use RL techniques to co-train a set of agents in a multi-agent system (Papoudakis et al., 2019).”
Comparative Evaluation of Multi-Agent Deep Reinforcement Learning Algorithms (2020, background): “Recent years have seen a plethora of new MARL algorithms which integrate deep learning techniques to aid learning (e.g. see survey [26]).”
Multi-agent deep reinforcement learning: a survey (2021, unknown): “A similar categorization proposed Papoudakis et al. (2019).”
Applications of reinforcement learning in energy systems (2021, background): “Despite several practical problems (e.g., non-stationarity issues [203], curse of dimensionality, and computational demand) associated with deep MARL, in several recent works, empirical success in complex multi-agent scenarios has been reported.”
Local Information Agent Modelling in Partially-Observable Environments (2021, methodology): “Multi-agent Reinforcement Learning (MARL): MARL algorithms are designed to train multiple agents to solve tasks in a shared environment [Hernandez-Leal et al., 2017, Papoudakis et al., 2019].”
Dealing with Non-Stationarity in MARL via Trust-Region Decomposition (2021, background): “…function r t ( o , a i ) and Lemma 1 and Lemma 2 indicate that the δ -stationarity is a reasonable mathematical description of the non-stationarity problem (Hernandez-Leal et al., 2017; Papoudakis et al., 2019; Padakandla, 2021), which can measure the non-stationarity of the learning procedure.”