Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharing

By Filippos Christianos, Georgios Papoudakis, Arrasy Rahman, Stefano V. Albrecht. 2021

176 citations recorded by the source corpus.

Sharing parameters in multi-agent deep reinforcement learning has played an essential role in allowing algorithms to scale to a large number of agents. Parameter sharing between agents significantly decreases the number of trainable parameters, shortening training times to tractable levels, and has been linked to more efficient learning. However, having all agents share the same parameters can also have a detrimental effect on learning. We demonstrate the impact of parameter sharing methods on training speed and converged returns, establishing that when applied indiscriminately, their effectiveness is highly dependent on the environment. We propose a novel method to automatically identify agents which may benefit from sharing parameters by partitioning them based on their abilities and goals. Our approach combines the increased sample efficiency of parameter sharing with the representational capacity of multiple independent networks to reduce training time and increase final returns.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tonghan Wang
2. Chongjie Zhang
3. Yu Wang
4. Chenghao Li
5. Qianchuan Zhao
6. Chao Yu
7. Jun Yang
8. Alexandre M. Bayen

In Control and Systems Engineering

1. Eugene Vinitsky
2. Alexandre M. Bayen
3. Chenghao Li
4. Jun Yang
5. Yi Wu
6. Qianchuan Zhao

In Reinforcement Learning

1. Tonghan Wang
2. Chongjie Zhang
3. Yu Wang
4. Alexandre M. Bayen
5. Chao Yu
6. Eugene Vinitsky
7. Yi Wu
8. Akash Velu

In Optimization & Training

1. Eugene Vinitsky
2. Alexandre M. Bayen
3. Chongjie Zhang
4. Tonghan Wang
5. Yi Wu
6. Yu Wang

Citations in the corpus

• The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games (2021, positive): “agents have identical observation and action spaces), we utilize parameter sharing; past works have shown that this improves the efficiency of learning [5, 32], which is also consistent with our empirical experiences.”
• Celebrating Diversity in Shared Multi-Agent Reinforcement Learning (2021, background): “[40] group agents during pre-training and force agents in the same group to use one policy.”