Predicting and Preventing Coordination Problems in Cooperative Q-learning Systems

By Nancy Fulda, Dan Ventura. 2007

65 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability (2017, background): “ILs, in particular, are susceptible to shadowed equilibria , where local observability and non-stationarity cause locally optimal actions to become a globally sub-optimal joint action (Fulda & Ventura, 2007).”
Decentralized Likelihood Quantile Networks for Improving Performance in Deep Multi-Agent Reinforcement Learning (2018, background): “In particular, with high probability, the agents will not converge to an optimal joint policy, but optimal independent policies under the effect of environment non-stationarity caused by other agents’ optimal independent policies (Fulda & Ventura, 2007).”
Negative Update Intervals in Deep Multi-Agent Reinforcement Learning (2018, background): “This can lead to action shadowing, where miscoordination due to sub-optimal joint-policies results in utility values of optimal actions being underestimated [5, 20].”
Independent reinforcement learners in cooperative Markov games: a survey regarding coordination problems (2012, methodology): “Our deﬁnition is closer in spirit to the one of Fulda and Ventura (2007).”
A survey and critique of multiagent deep reinforcement learning (2018, background): “’ joint policy ˇ i(s;a i) can be non-stationary, thus becoming the parameter of the best response function. There are other common problems in MAL, including convergence [2, 49, 50], action shadowing [24, 51], the curse of dimensionality [5], and multiagent credit assignment [23]. Describing each pr”
Multi-agent deep reinforcement learning: a survey (2021, background): “However, it has been shown that agents can converge, despite a high degree of randomness in action selection, to sub-optimal solutions or can get stuck between different solutions (Wiegand 2004).”
An exploration strategy for non-stationary opponents (2017, background): “This effect is known as shadowing(3) [20] and can only be avoided by continuously checking far visited states.”
Likelihood Quantile Networks for Coordinating Multi-Agent Reinforcement Learning (2018, methodology): “Unfortunately, with high probability, the decentralized learning agents will not converge to an optimal joint policy, but only optimal independent policies under the effect of environment non-stationarity caused by other agents’ optimal independent policies [10].”
Learning to compete, coordinate, and cooperate in repeated games using reinforcement learning (2011, background): “This allows the agent to avoid action shadowing ( Fulda and Ventura 2007; Stimpson et al. 2001), in which the estimated utility for taking an action corresponding to a good outcome is low because the other possible outcomes for taking that action are low.”
A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity (2017, background): “This effect is known as shadowing (Fulda and Ventura, 2007) or observationally equivalent models (Doshi and Gmytrasiewicz, 2006).”
Dynamic analysis of multiagent Q-learning with ε-greedy exploration (2009, background): “Fulda and Ventura (2007), for example, presented a set of conditions, on the environment and the payo. tables, which are su.”
Lenient Learning in Independent-Learner Stochastic Cooperative Games (2016, methodology): “Lauer and Riedmiller introduced Distributed Q-learning (2000), which addresses two major problems mentioned in (Fulda and Ventura, 2007).”
Theoretical Advantages of Lenient Learners: An Evolutionary Game Theoretic Perspective (2008, background): “For example, Fulda and Ventura (2007) attribute the convergence to suboptimal solutions to what they term action shadowing.”
The world of independent learners is not markovian (2011, background): “Indeed, in cooperative multi-agent systems, theoretical and experimental works showed that only few couplings, as action shadowing, raise convergence issues [11,18].”
Learning the IPA market with individual and social rewards (2007, unknown): “According to Fulda and Ventura [13], from the sys-tem perspective, an optimal solution would be any one which is a NE and Pareto-Optimal (PO).”