Learning in Multi-agent Systems with Sparse Interactions by Knowledge Transfer and Game Abstraction

By Yujing Hu, Yang Gao, Bo An. 2015

19 citations recorded by the source corpus.

In many multi-agent systems, the interactions between agents are sparse and exploiting interaction sparseness in multi-agent reinforcement learning (MARL) can improve the learning performance. Also, agents may have already learnt some single-agent knowledge (e.g., local value function) before the multi-agent learning process. In this work, we investigate how such knowledge can be utilized to learn better policies in multi-agent systems with sparse interactions. We adopt game theory-based MARL as the basic learning approach since it can coordinate agents better. We contribute three knowledge transfer mechanisms. The first one is value function transfer, which directly transfers agents' local value functions to the learning algorithm. The second one is selective value function transfer, which only transfers the value functions in states where the environmental dynamics change slightly. The last mechanism is model transfer-based game abstraction, which further improves the former two mechanisms by abstracting the one-shot game in each state and reducing equilibrium computation. Experimental results in benchmarks show that with the three knowledge transfer mechanisms, all of the tested game theory-based MARL algorithms are drastically improved and also achieve better asymptotic performance than the state-of-the-art algorithm CQ-learning.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Y. Liu
2. Yujing Hu
3. Yang Gao
4. Yingfeng Chen
5. Changjie Fan
6. Weixun Wang
7. Jianye Hao
8. Shimon Whiteson

In Computational Theory and Mathematics

1. Y. Liu
2. Yujing Hu
3. Yang Gao
4. Jianye Hao

Citations in the corpus

• Value Function Transfer for Deep Multi-Agent Reinforcement Learning Based on N-Step Returns (2019, methodology): “A commonly used technique for defin-ing MDP similarity is bisimulation metric [Hu et al. , 2015; Song et al. , 2016; Ferns et al. , 2004], which exactly captures the property of rewards and state transitions in an MDP.”
• Multi-Agent Game Abstraction via Graph Attention Neural Network (2019, methodology): “…techniques such as game abstraction and knowledge transfer to help with speeding up multi-agent re-inforcement learning(Guestrin, Lagoudakis, and Parr 2002; Kok and Vlassis 2004; De Hauwere, Vrancx, and Now ´ e 2010; Melo and Veloso 2011; Hu, Gao, and An 2015; Yu et al. 2015; Liu et al. 2019).”