Mean Field Multi-Agent Reinforcement Learning

By Yaodong Yang, Rui Luo, Minne Li, M. Zhou, Weinan Zhang, Jun Wang. 2018

717 citations recorded by the source corpus.

Existing multi-agent reinforcement learning methods are limited typically to a small number of agents. When the agent number increases largely, the learning becomes intractable due to the curse of the dimensionality and the exponential growth of user interactions. In this paper, we present Mean Field Reinforcement Learning where the interactions within the population of agents are approximated by those between a single agent and the average effect from the overall population or neighboring agents; the interplay between the two entities is mutually reinforced: the learning of the individual agent's optimal policy depends on the dynamics of the population, while the dynamics of the population change according to the collective patterns of the individual policies. We develop practical mean field Q-learning and mean field Actor-Critic algorithms and analyze the convergence of the solution. Experiments on resource allocation, Ising model estimation, and battle game tasks verify the learning effectiveness of our mean field approaches in handling many-agent interactions in population.

Subfields: Artificial Intelligence, Management Science and Operations Research, Statistical and Nonlinear Physics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Statistical and Nonlinear Physics

In Reinforcement Learning

Citations in the corpus

Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2020, background): “…Pong (Tampuu et al., 2017), Keepaway Soccer (Stone et al., 2005), or simple gridworld-like environments (Lowe et al., 2017; Leibo et al., 2017; Yang et al., 2018; Zheng et al., 2017), there are currently no challenging standard testbeds for centralised training with decentralised execution…”
Wireless Network Intelligence at the Edge (2018, methodology): “To fill this gap, in the context of large-scale MARL, the population dynamics of the MFG has been inferred using an inverse RL method in [169].”
Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharing (2021, methodology): “, 2018), or Mean Field RL (Yang et al., 2018) share the parameters of neural networks similarly to our FuPS and FuPS+id baselines.”
QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning (2019, background): “Multi-domain Gaussian Squeeze (MGS) Gaussian Squeeze (GS) (HolmesParker et al., 2014) is a simple nonmonotonic multi-agent resource allocation problem, used in other papers, e.g., Yang et al. (2018) and Chen et al. (2018).”
The StarCraft Multi-Agent Challenge (2019, background): “, 2005), or simple gridworld-like environments (Lowe et al., 2017; Leibo et al., 2017; Yang et al., 2018; Zheng et al., 2017).”
From Few to More: Large-scale Dynamic Multiagent Curriculum Learning (2019, background): “Furthermore, in large-scale multiagent systems, the dynamics and stochasticity of the environment become more complex, which makes it more challenging to achieve coordination among agents (Singh, Jain, and Sukhbaatar 2019; Yang et al. 2018a; Jiang and Lu 2018).”
Action Semantics Network: Considering the Effects of Actions in Multiagent Systems (2019, background): “Recently, Yang et al. (2018) proposed applying mean-field theory (Stanley, 1971) to solve large-scale multiagent learning problems.”
Factorized Q-learning for large-scale multi-agent systems (2018, methodology): “This is also the case for the recently emerged mean-ﬁeld Q-learning (MF-Q) algorithm (Yang et al. 2018) which equips each agent with one Q-function and approximates it by the average effects in the agent’s neighborhood.”
Learning Attentional Communication for Multi-Agent Cooperation (2018, background): “Mean Field [24] takes as input the observation and mean action of neighboring agents to make the decision.”
A survey and critique of multiagent deep reinforcement learning (2018, background): “Our conclusions of this work are that while the number of works in DRL and MDRL are notable and represent important milestones for AI, at the same time we acknowledge there are also open questions in both (deep) single-agent learning [38, 298, 362, 79] and multiagent learning [363, 364, 365, 366, 36”
Graph Convolutional Reinforcement Learning for Multi-Agent Cooperation (2018, methodology): “Information sharing indeed helps as DGN-M, MFQ, and CommNet all outperform DQN.”
Off-Policy Multi-Agent Decomposed Policy Gradients (2020, background): “Another line of research focuses on fully decentralized actor-critic learning (Macua et al., 2017; Zhang et al., 2018; Yang et al., 2018; Cassano et al., 2018; Suttle et al., 2019; Zhang & Zavlanos, 2019).”
Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms (2019, methodology): “This mean-field view of multi-agent systems has been approached by the mean-field games (MFGs) model [165, 166, 167, 168, 169], the team model with mean-field sharing [170, 171], and the game model with mean-field actions [172].”
Multi-agent deep reinforcement learning: a survey (2021, methodology): “Another work by Yang et al. (2018a) used Bayesian techniques to detect opponent strategies in competitive games.”
Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning (2020, neutral): “…experiments: (1) the MADDPG algorithm (Lowe et al., 2017) with its original architecture ( MADDPG ); (2) the provably-converged mean-ﬁeld algorithm (Yang et al., 2018) ( mean-ﬁeld ); (3) the MADDPG algorithm with our population-invariant architecture ( Att-MADDPG ); (4) the vanilla population…”
A review of cooperative multi-agent deep reinforcement learning (2019, background): “[161] considers the multi-agent RL problems in the case that there exists a huge number of agents collaborating or competing with each other to optimize some specific long-term rewards.”