Influence-Based Multi-Agent Exploration

By Tonghan Wang, Jianhao Wang, Yi Wu, Chongjie Zhang. 2019

160 citations recorded by the source corpus.

Intrinsically motivated reinforcement learning aims to address the exploration challenge for sparse-reward tasks. However, the study of exploration methods in transition-dependent multi-agent settings is largely absent from the literature. We aim to take a step towards solving this problem. We present two exploration methods: exploration via information-theoretic influence (EITI) and exploration via decision-theoretic influence (EDTI), by exploiting the role of interaction in coordinated behaviors of agents. EITI uses mutual information to capture influence transition dynamics. EDTI uses a novel intrinsic reward, called Value of Interaction (VoI), to characterize and quantify the influence of one agent's behavior on expected returns of other agents. By optimizing EITI or EDTI objective as a regularizer, agents are encouraged to coordinate their exploration and learn policies to optimize team performance. We show how to optimize these regularizers so that they can be easily integrated with policy gradient reinforcement learning. The resulting update rule draws a connection between coordinated exploration and intrinsic reward distribution. Finally, we empirically demonstrate the significant strength of our method in a variety of multi-agent scenarios.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Safety Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Safety Research

Citations in the corpus

Celebrating Diversity in Shared Multi-Agent Reinforcement Learning (2021, background): “EITI and EDTI consider pairwise mutual inﬂuence to encourage the interdependence between agents.”
RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, methodology): “However, learning roles from scratch might not be easier than learning without roles, as directly ﬁnding an optimal decomposition suffers from the same problem as other CTDE learning methods – searching in the large joint space with substantial exploration (Wang et al., 2020c).”
ROMA: Multi-Agent Reinforcement Learning with Emergent Roles (2020, background): “Roles can change adaptively and will aggregate according to responsibilities that are compatible with individual characteristics, such as location, agent type, health, etc. et al., 2019; Wang et al., 2020b; Baker et al., 2020).”
The Emergence of Individuality in Multi-Agent Reinforcement Learning (2020, methodology): “Then, we compare EOI against EDTI [27] to verify the advantages of individuality with sparse reward, and against ROMA [12] and [10, 11] (denoted as HC and HC-simul respectively) to investigate the advantages of individuality over emergent roles and diversity.”
Off-Policy Multi-Agent Decomposed Policy Gradients (2020, background): “The centralized training with decentralized execution (CTDE) paradigm (Foerster et al., 2016; Wang et al., 2020d) has recently attracted attention for its ability to address non-stationarity while maintaining decentralized execution.”
Towards Understanding Linear Value Decomposition in Cooperative Multi-Agent Q-Learning (2020, background): “[15] Tonghan Wang, Jianhao Wang, Yi Wu, and Chongjie Zhang.”
Multi-agent deep reinforcement learning: a survey (2021, background): “…by applying intrinsic motivation (Jaques et al. 2018, 2019), by making the communication protocol more robust or scalable (Kim et al. 2019; Singh et al. 2019), and maximizing the utility of the communication through efficient encoding (Celikyilmaz et al. 2018; Li et al. 2019b; Wang et al. 2020c).”