Exploration with Unreliable Intrinsic Reward in Multi-Agent Reinforcement Learning

By Wendelin Böhmer, Tabish Rashid, Shimon Whiteson. 2019

26 citations recorded by the source corpus.

This paper investigates the use of intrinsic reward to guide exploration in multi-agent reinforcement learning. We discuss the challenges in applying intrinsic reward to multiple collaborative agents and demonstrate how unreliable reward can prevent decentralized agents from learning the optimal policy. We address this problem with a novel framework, Independent Centrally-assisted Q-learning (ICQL), in which decentralized agents share control and an experience replay buffer with a centralized agent. Only the centralized agent is intrinsically rewarded, but the decentralized agents still benefit from improved exploration, without the distraction of unreliable incentives.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Computational Theory and Mathematics

Citations in the corpus

Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge? (2020, methodology): “…independent learning algorithms IAC and IQL, on a number of both IPPO with policy and value clipping [6] IPPO without value clipping [5] IPPO without policy clipping [5] IPPO with neither policy nor value clipping [6] IPPO without clipping, with low learning rate [4] 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.”
Weighted QMIX: Expanding Monotonic Value Function Factorisation (2020, background): “Bohmer et al. [3] learn a centralised joint action Q -”
Deep Coordination Graphs (2019, background): “Böhmer et al. (2019) present another approach to decentralize a centralized value function, which is locally maximized by coordinate ascent and decentralized by training IQL agents from the same replay buffer.”