Lenient Learning in Independent-Learner Stochastic Cooperative Games

By Ermo Wei, Sean Luke. 2016

107 citations recorded by the source corpus.

Subfields: Reinforcement Learning, Game & Decision Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Game & Decision Theory

Citations in the corpus

Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2020, background): “Wei and Luke (2016) show that Leniency outperforms Hysteretic Q -learning in cooperative stochastic games, and Palmer et al. (2018) extend Leniency to the deep RL setting and show beneﬁts over Hysteretic and fully Independent Q -learners on deterministic and stochastic gridworlds with two agents.”
Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge? (2020, background): “…joint learning reduces or removes issues surrounding partial observability and environment non-stationarity, it must cope with joint action spaces that grow exponentially with respect to the number of agents, as well as a variety of learning pathologies that can result in suboptimal policies [32].”
QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning (2019, background): “Wei et al. (2018) proposes MultiAgent Soft Q-learning in continuous action spaces to tackle the relative overgeneralization problem (Wei & Luke, 2016) and achieves better coordination.”
QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2018, background): “eretic Q-learning in a multi-task deep RL setting. Panait et al. (2008) introduce Leniency which ignores updates that decrease Q-value estimates with a probability that is decreasing during training. Wei and Luke (2016) show that Leniency outperforms Hysteretic Q-learning in cooperative stochastic g”
The Representational Capacity of Action-Value Networks for Multi-Agent Reinforcement Learning (2019, background): “In the Climb Game [31], each agent has three local actions A i = { a 0 ,a 1 ,a 2 } .”
Decentralized Likelihood Quantile Networks for Improving Performance in Deep Multi-Agent Reinforcement Learning (2018, critical): “Therefore, these methods can mistake a high reward resulting from environment stochasticity as a successful cooperation (Wei & Luke, 2016).”
Negative Update Intervals in Deep Multi-Agent Reinforcement Learning (2018, methodology): “For example, if both lines end with red dots in the top corner of a simplex, then the two agents are predominately producing trajectories τ ∈ TA, and have converged upon the optimal Nash Equilibrium (A,A).”
Lenient Multi-Agent Deep Reinforcement Learning (2017, methodology): “Similar to the hysteretic approach, lenient agents initially adopt an optimistic disposition, before gradually transforming into average reward learners [35].”
Multiagent Soft Q-Learning (2018, background): “When the information is limited for learners in cooperative games, as is the case with independent learners, a pathology called relative overgeneralization can arise (Wei and Luke 2016).”
FACMAC: Factored Multi-Agent Centralised Policy Gradients (2020, methodology): “The gradient estimator used in MADDPG is also known to be vulnerable to relative overgeneralisation [44].”
Action Branching Architectures for Deep Reinforcement Learning (2017, background): “This could be due to any of the several well-known issues for independent fully-cooperative learning agents: Pareto-selection, non-stationarity, stochasticity, alter-exploration, and shadowed equilibria [Matignon et al., 2012, Wei and Luke, 2016].”
Weighted Double Deep Multiagent Reinforcement Learning in Stochastic Cooperative Environments (2018, background): “…fold the average temperature ¯ T ( s (cid:48) ) = 1 | A | a i ∈ A T ( s (cid:48) , a i ) for next state s (cid:48) into the temperature that is being decayed for ( s t , a t ) [ Wei and Luke, 2016 ] , as below: (7) where η is a constant controlling the extent that ¯ T ( s (cid:48) ) is folded in.”
Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learning (2019, unknown): “The pathology of ﬁnding a suboptimal Nash equilibrium is also called relative over-generalization (Wei & Luke, 2016).”
Multi-agent deep reinforcement learning: a survey (2021, methodology): “This phenomenon arises since each agent’s policy performs relatively well when paired with arbitrary actions from other agents (Panait et al. 2006; Wei and Luke 2016; Wiegand 2004).”
Likelihood Quantile Networks for Coordinating Multi-Agent Reinforcement Learning (2018, background): “Therefore, these methods can mistake a high reward resulting from stochasticity as a successful cooperation [32].”
FOP: Factorizing Optimal Joint Policy of Maximum-Entropy Multi-Agent Reinforcement Learning (2021, background): “…of the actors of MADDPG and MAAC has large variance due to the centralized critic (Wang et al., 2020), which is also The pathology of ﬁnding a sub-optimal solu-tion is also called relative overgeneralization (Wei & Luke, 2016; Castellini et al., 2019), which is discussed further in Appendix D.2.”