A Deep Policy Inference Q-Network for Multi-Agent Systems

By Zhang-Wei Hong, Shih-Yang Su, Tzu-Yun Shann, Yi-Hsiang Chang, Chun-Yi Lee. 2017

93 citations recorded by the source corpus.

We present DPIQN, a deep policy inference Q-network that targets multi-agent systems composed of controllable agents, collaborators, and opponents that interact with each other. We focus on one challenging issue in such systems---modeling agents with varying strategies---and propose to employ "policy features'' learned from raw observations (e.g., raw images) of collaborators and opponents by inferring their policies. DPIQN incorporates the learned policy features as a hidden vector into its own deep Q-network (DQN), such that it is able to predict better Q values for the controllable agents than the state-of-the-art deep reinforcement learning models. We further propose an enhanced version of DPIQN, called deep recurrent policy inference Q-network (DRPIQN), for handling partial observability. Both DPIQN and DRPIQN are trained by an adaptive training procedure, which adjusts the network's attention to learn the policy features and its own Q-values at different phases of the training process. We present a comprehensive analysis of DPIQN and DRPIQN, and highlight their effectiveness and generalizability in various multi-agent settings. Our models are evaluated in a classic soccer game involving both competitive and collaborative scenarios. Experimental results performed on 1 vs. 1 and 2 vs. 2 games show that DPIQN and DRPIQN demonstrate superior performance to the baseline DQN and deep recurrent Q-network (DRQN) models. We also explore scenarios in which collaborators or opponents dynamically change their policies, and show that DPIQN and DRPIQN do lead to better overall performance in terms of stability and mean scores.

Subfields: Artificial Intelligence, Economics and Econometrics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Economics and Econometrics

Citations in the corpus

Continuous Coordination As a Realistic Scenario for Lifelong Learning (2021, background): “Hong et al. (2017) aim to learn models of opponents by learning policy features from raw observations of other agents.”
A survey and critique of multiagent deep reinforcement learning (2018, background): “Albrecht and Stone [11] presented a thorough survey in this topic and we have found many works that fit into this category in the MDRL setting, some taking inspiration from DRL [169, 170, 171], and others from MAL [172, 173, 64, 174, 175].”
Relational Forward Models for Multi-Agent Learning (2018, positive): “In learning-based systems, there have been some successes by introducing a centralized controller (D’Andrea, 2012; Foerster et al., 2016; 2017; Hong et al., 2017; Lowe et al., 2017).”
Actor-Critic Policy Optimization in Partially Observable Multiagent Environments (2018, methodology): “There are also methods that attempt to model the opponents [36, 30, 4].”
Multi-agent deep reinforcement learning: a survey (2021, background): “…can be found in cooperative tasks (Barde et al. 2019; Tacchetti et al. 2019; Zheng et al. 2018b) with imitation learning (Grover et al. 2018; Le et al. 2017), in social dilemmas (Jaques et al. 2019; Letcher et al. 2019), and by predicting behaviors from observations (Hong et al. 2017; Hoshen 2017).”
Reinforcement Learning and Its Applications in Modern Power and Energy Systems: A Review (2020, background): “Recent studies also extend RNNs to the MADRL setting to solve the challenges of partially observable Markov games [40], [41].”
A Q-values Sharing Framework for Multi-agent Reinforcement Learning under Budget Constraint (2020, background): “By contrast, we consider that agents are decentralized in both training and execution, which is convenient for agents joining and leaving [8, 9, 10].”
Promoting Coordination through Policy Regularization in Multi-Agent Deep Reinforcement Learning (2019, methodology): “In general, this type of behavior prediction can be referred to as agent modelling (or opponent modelling) and has been used in previous work to enrich representations [12, 13], to stabilise the learning dynamics [10] or to classify the opponent’s play style [29].”
Deep Reinforcement Learning for Multiagent Systems: A Review of Challenges, Solutions, and Applications (2018, unknown): “[61] introduced a deep policy inference”