Dynamic analysis of multiagent Q-learning with ε-greedy exploration

By Eduardo Rodrigues Gomes, Ryszard Kowalczyk. 2009

146 citations recorded by the source corpus.

The development of mechanisms to understand and model the expected behaviour of multiagent learners is becoming increasingly important as the area rapidly find application in a variety of domains. In this paper we present a framework to model the behaviour of Q-learning agents using the ε-greedy exploration mechanism. For this, we analyse a continuous-time version of the Q-learning update rule and study how the presence of other agents and the ε-greedy mechanism affect it. We then model the problem as a system of difference equations which is used to theoretically analyse the expected behaviour of the agents. The applicability of the framework is tested through experiments in typical games selected from the literature.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

A Multi-Agent Framework for Packet Routing in Wireless Sensor Networks (2015, methodology): “In Line 3, π is calculated using Equation (7) for each available action, a, which is the -greedy exploration method [33], where 0 < < 1 is a small positive number and n is the number of available actions of a sensor.”
Independent reinforcement learners in cooperative Markov games: a survey regarding coordination problems (2012, background): “In (Gomes & Kowalczyk, 2009; Wunder, Littman, & Babes, 2010), the authors analyse a continuous-time version of decentralized Q-learning.”
Dynamic Safe Interruptibility for Decentralized Multi-Agent Reinforcement Learning (2017, background): “Besides, convergence to a Nash equilibrium situation where no agent has interest in changing policies is generally not guaranteed even for suboptimal equilibria on simple games [27, 18].”
Classes of Multiagent Q-learning Dynamics with epsilon-greedy Exploration (2010, methodology): “The authors are supported via NSF HSD-0624191 and thank S. Singh and M. Kaisers for discussions.”
Learning-Based Computation Offloading for IoT Devices With Energy Harvesting (2017, methodology): “To make a tradeoff between exploitation and exploration, the IoT device selects the offloading policy, a(k) ∈ A, based on the output of the CNN according to the ε-greedy policy [35], and then sends x(k)C(k) data to edge device i.”
Hierarchical Reinforcement Learning: A Survey (2015, methodology): “C. Q-Learning Q-learning is one of the RL algorithms that has been successfully used in many domains such as: face recognition, simple toys, web-based education and many others [8].”
A Tutorial on Ultrareliable and Low-Latency Communications in 6G: Integrating Domain Knowledge Into Deep Learning (2021, methodology): “There are many sophisticated methods to balance exploitation and exploration, such as ε -greedy methods [241], optimistic initial values [68], and the Bayesian sampling approach [242].”
Artificial Intelligence, Algorithmic Pricing, and Collusion (2020, unknown): “Second, for the ε -greedy model of exploration used in this paper, the deterministic dynamics is complicated by the jumps that occur when the Q-values of the best and second-best actions cross, which give raise to hybrid dynamical systems (Gomes and Kowalczyk, 2009; Wunder et al. 2010).”