Hysteretic q-learning :an algorithm for decentralized reinforcement learning in cooperative multi-agent teams

By Laëtitia Matignon, Guillaume J. Laurent, Nadine Le Fort-Piat. 2007

262 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Control and Systems Engineering, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2020, background): “Matignon et al. (2007) introduce Hysteretic Q - learning which instead uses a smaller learning rate for decreasing Q -value estimates, which is slightly more robust to stochasticity and the presence of multiple optimal joint actions.”
Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability (2017, background): “Hysteretic Q-learning (Matignon et al., 2007), instead, uses the insight that low returns may also be caused by domain stochasticity, which should not be ignored.”
Weighted QMIX: Expanding Monotonic Value Function Factorisation (2020, background): “Optimistic Weighting uses a smaller weighting when decreasing Qtot estimates, similarly to Hysteretic Q-learning [17] which uses a smaller learning rate when decreasing value estimates of independent learners.”
Decentralized Likelihood Quantile Networks for Improving Performance in Deep Multi-Agent Reinforcement Learning (2018, background): “Two learning rates α and β , named increase rate and decrease rate, are respectively used for updating overestimated and underestimated TD error δ : Hysteretic Deep Q-Network (HDQN) (Omidshaﬁei et al., 2017) applies HQL to DQN.”
Negative Update Intervals in Deep Multi-Agent Reinforcement Learning (2018, methodology): “Hysteretic Q-Learning is a form of optimistic learning that helps agents overcome miscoordination [15].”
Deep Reinforcement Learning for Swarm Systems (2018, background): “Omidshaﬁei et al. [2017] investigate hysteretic Q-learning [Matignon et al., 2007] and distillation [Rusu et al., 2015].”
Lenient Multi-Agent Deep Reinforcement Learning (2017, methodology): “Two well researched approaches used to help parallel reinforcement learning agents overcome the moving target problem in these domains include hysteretic Q-learning [17] and leniency [22].”
Independent reinforcement learners in cooperative Markov games: a survey regarding coordination problems (2012, methodology): “To avoid this overestimation in stochastic games, we introduced hysteretic learners (Matignon et al., 2007).”
Deep Reinforcement Learning for Multiagent Systems: A Review of Challenges, Solutions, and Applications (2018, background): “ers [62], DRQNs [44], distillation [63], and concurrent experience replay trajectories (CERTs), which are a decentralized extension of experience replay strategy proposed”
Lenient Learning in Independent-Learner Stochastic Cooperative Games (2016, background): “Building on Distributed Q-learning, Hysteretic Q-learning (Matignon et al., 2007) uses two different learning rates to address stochasticity.”
Multi-Agent Reinforcement Learning: A Review of Challenges and Applications (2021, methodology): “The multi-agent setting is addressed via independent learner agents using Q-learning to maximize the future rewards, defined as the difference between the data throughput and the cost of power consumption.”
Weighted Double Deep Multiagent Reinforcement Learning in Stochastic Cooperative Environments (2018, background): “Approaches like [Bloembergen et al., 2011; Matignon et al., 2007; Matignon et al., 2012; Panait et al., 2006; Wei and Luke, 2016] have been proposed by extending Qlearning to address the coordination problems in cooperative multiagent systems.”
CESMA: Centralized Expert Supervises Multi-Agents (2019, background): “…Q -learning to the multi-agent setting: Distributed Q-Learning (Lauer & Ried-miller, 2000) updates Q-values only when they increase,and updates the policy only for actions that are not greedy with respect to the Q -values, and Hysteretic Q -Learning (Matignon et al., 2007) provides a modiﬁcation.”
Learning Transferable Cooperative Behavior in Multi-Agent Teams (2019, background): “Other works like [19, 24] present a formalization of multi-agent decentralized RL under partial observability and also address learning stabilisation, and present results on normal form games or predator-prey games with up to 3 agents.”
Multi-agent deep reinforcement learning: a survey (2021, background): “Stochasticity, relative over-gener-alization, and other pathologies such as non-stationarity and the alter-exploration problem led to new branches of research including hysteretic learning (Matignon et al. 2007) and leniency (Potter and De Jong 1994).”
Two Body Problem: Collaborative Visual Task Completion (2019, background): “Similarly, a variety of settings from multiple cooperative agents to multiple competitive ones have been investigated [51, 65, 57, 11, 63, 35, 56, 29, 61].”
Quo vadis artificial intelligence? (2022, background): “A typical example is the application of collaborative robot control [64].”