Value Function Transfer for Deep Multi-Agent Reinforcement Learning Based on N-Step Returns

By Y. Liu, Yujing Hu, Yang Gao, Yingfeng Chen, Changjie Fan. 2019

34 citations recorded by the source corpus.

Many real-world problems, such as robot control and soccer game, are naturally modeled as sparse-interaction multi-agent systems. Reutilizing single-agent knowledge in multi-agent systems with sparse interactions can greatly accelerate the multi-agent learning process. Previous works rely on bisimulation metric to define Markov decision process (MDP) similarity for controlling knowledge transfer. However, bisimulation metric is costly to compute and is not suitable for high-dimensional state space problems. In this work, we propose more scalable transfer learning methods based on a novel MDP similarity concept. We start by defining the MDP similarity based on the N-step return (NSR) values of an MDP. Then, we propose two knowledge transfer methods based on deep neural networks called direct value function transfer and NSR-based value function transfer. We conduct experiments in image-based grid world, multi-agent particle environment (MPE) and Ms. Pac-Man game. The results indicate that the proposed methods can significantly accelerate multi-agent reinforcement learning and meanwhile get better asymptotic performance.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Shimon Whiteson
2. Christian Schröder de Witt
3. Tabish Rashid
4. Gregory Farquhar
5. Jakob N. Foerster
6. Jianye Hao
7. Chongjie Zhang
8. Mikayel Samvelyan

In Computational Theory and Mathematics

1. Shimon Whiteson
2. Tabish Rashid
3. Christian Schröder de Witt
4. Jakob N. Foerster
5. Mikayel Samvelyan
6. Gregory Farquhar
7. Y. Liu
8. Yujing Hu

In Reinforcement Learning

1. Shimon Whiteson
2. Christian Schröder de Witt
3. Tabish Rashid
4. Gregory Farquhar
5. Jakob N. Foerster
6. Chongjie Zhang
7. Jianye Hao
8. Mikayel Samvelyan

In Meta & Continual Learning

1. Shimon Whiteson
2. Yang Gao

Citations in the corpus

• Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2020, methodology): “Liu et al. (2019) investigate transfer learning algorithms in a multi-agent setting using QMIX as a base.”
• QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2018, neutral): “nt a larger class of joint action-value functions than QMIX. Since they also use non-negative weights in their mixing network, SMIX() can represent exactly the same class of value functions as QMIX. Liu et al. (2019) investigate transfer learning algorithms in a multi-agent setting using QMIX as a b”
• Multi-Agent Game Abstraction via Graph Attention Neural Network (2019, methodology): “…techniques such as game abstraction and knowledge transfer to help with speeding up multi-agent re-inforcement learning(Guestrin, Lagoudakis, and Parr 2002; Kok and Vlassis 2004; De Hauwere, Vrancx, and Now ´ e 2010; Melo and Veloso 2011; Hu, Gao, and An 2015; Yu et al. 2015; Liu et al. 2019).”