Negative Update Intervals in Deep Multi-Agent Reinforcement Learning

By Gregory Palmer, Rahul Savani, Karl Tuyls. 2018

28 citations recorded by the source corpus.

In Multi-Agent Reinforcement Learning (MA-RL), independent cooperative learners must overcome a number of pathologies to learn optimal joint policies. Addressing one pathology often leaves approaches vulnerable towards others. For instance, hysteretic Q-learning \citematignon2007hysteretic addresses miscoordination while leaving agents vulnerable towards misleading stochastic rewards. Other methods, such as leniency, have proven more robust when dealing with multiple pathologies simultaneously \citeJMLR:v17:15-417. However, leniency has predominately been studied within the context of strategic form games (bimatrix games) and fully observable Markov games consisting of a small number of probabilistic state transitions. This raises the question of whether these findings scale to more complex domains. For this purpose we implement a temporally extend version of the Climb Game \citeclaus1998dynamics, within which agents must overcome multiple pathologies simultaneously, including relative overgeneralisation, stochasticity, the alter-exploration and moving target problems, while learning from a large observation space. We find that existing lenient and hysteretic approaches fail to consistently learn near optimal joint-policies in this environment. To address these pathologies we introduce Negative Update Intervals-DDQN (NUI-DDQN), a Deep MA-RL algorithm which discards episodes yielding cumulative rewards outside the range of expanding intervals. NUI-DDQN consistently gravitates towards optimal joint-policies in our environment, overcoming the outlined pathologies.

Subfields: Artificial Intelligence, Management Science and Operations Research, Safety Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Safety Research

In Reinforcement Learning

Citations in the corpus

Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2020, background): “Palmer et al. (2019) maintain a learned interval whose lower bound is approximately the minimum cumulative reward received when all agents are coordinating.”
A survey and critique of multiagent deep reinforcement learning (2018, methodology): “nt scenarios. However, learning in multiagent environments is fundamentally more dicult due to non-stationarity, the increase of dimensionality, and the creditassignment problem, among other factors [1, 5, 10, 147, 241, 361, 97]. This survey provides broad overview of recent works in the emerging ar”
Neural Replicator Dynamics (2019, background): “Most other works that focus on nonstationary multiagent learning are limited to repeated and stochastic games [1, 15, 24, 46, 47].”
Multi-agent deep reinforcement learning: a survey (2021, methodology): “Extensions to the deep Q-network can be seen in additional mechanisms used for the experience replay (Palmer et al. 2019), the utilization of specialized estimators (Zheng et al. 2018a) and the use of implicit quantile networks (Lyu and Amato 2020).”
Likelihood Quantile Networks for Coordinating Multi-Agent Reinforcement Learning (2018, background): “…policies, especially explorations, but also adaptively schedules the amount of optimism applied to each training sample based on estimated value maturity, achieving improved performance without hyper-parameter interventions [26, 28], scheduling tables [28] or specialized experience buffers [27].”