Friend-or-Foe Q-learning in General-Sum Games

By Michael L. Littman. 2001

561 citations recorded by the source corpus.

Subfields: Reinforcement Learning, Game & Decision Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Game & Decision Theory

Citations in the corpus

Value-Decomposition Networks For Cooperative Multi-Agent Learning (2017, background): “As is standard in MARL, the underlying environment is modeled as a Markov game where actions are chosen and executed simultaneously, and new observations are perceived simultaneously as a result of a transition to a new state (Littman, 1994, 2001; Hu and Wellman, 2003; Busoniu et al., 2008).”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “In these settings, even the correct solution concept can be difficult to define (Kim et al., 2022; Littman, 2001; Wang and Sandholm, 2002; Bowling, 2004; Greenwald and Hall, 2003; Zinkevich, Greenwald, and Littman, 2006; Letcher, Foerster, Balduzzi, Rocktäschel, and Whiteson, 2019).”
Multi-agent Reinforcement Learning in Sequential Social Dilemmas (2017, background): “The generalsum case is significantly more challenging [31], and algorithms either have strong assumptions or need to either track several different potential equilibria per agent [32, 33], model other players to simplify the problem [34], or must find a Figure 3: Left: Gathering.”
Empirically Evaluating Multiagent Learning Algorithms (2014, methodology): “Foe-Q is the same as minimax-Q .”
Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization (2021, background): “In RL, there are works on learning when to cooperate/compete (Littman, 2001; Peysakhovich & Lerer, 2018a; Kleiman-Weiner et al., 2016; Woodward et al., 2019; McKee et al., 2020), which is a special case of ours, or learning robust policies (Li et al., 2019; Shen & How, 2019; Hu et al., 2020), which…”
Learning with Opponent-Learning Awareness (2017, background): “Seminal work includes the family of WoLF algorithms (Bowling and Veloso, 2002), which achieve convergence by using different learning rates depending on whether an agent is winning or losing, Joint-action-learners, (Claus and Boutilier, 1998; Banerjee and Sen, 2007) and friend-or-foe learning (Littm”
Factorized Q-learning for large-scale multi-agent systems (2018, methodology): “For example, Nash-Q [5] and Friend-or-Foe [9] extend the classic Q-learning [30] by encoding the interactions between environment and agents in a so-called Nash Q function.”
Multiagent Learning: Basics, Challenges, and Prospects (2012, methodology): “Littman tackled these restrictive conditions of Nash-Q and introduced Friend or Foe Q-learning (Littman 2001), which converges to Nash equilibrium with fewer restrictions than Nash-Q.”
A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning (2017, background): “use adaptive learning rates [12], adjust the frequencies of updates [48, 81], or dynamically respond to the other agents actions online [63, 50].”
Multiagent Soft Q-Learning (2018, methodology): “Many methods have been proposed for equilibrium learning (Littman 1994; 2001; Hu and Wellman 2003; Greenwald, Hall, and Serrano 2003), where the agents are trying to learn policies that satisfy some equilibrium concept from Game Theory.”
A survey and critique of multiagent deep reinforcement learning (2018, methodology): “This concept has been explored in seminal MAL algorithms like Nash-Q learning [149] and Minimax-Q learning [198,199].”
If multi-agent learning is the answer, what is the question? (2007, background): “Later work (such as the joint-action learners in [14] and the Friend-or-Foe Q algorithm in [37]) proposed other update rules for the Q and V functions focusing on the special case of common-payoff (or ‘team’) games.”
Learning Zero-Sum Simultaneous-Move Markov Games Using Function Approximation and Correlated Equilibrium (2020, methodology): “In particular, under the tabular setting, the work in Littman (1994, 2001a,b); Greenwald et al. (2003); Hu and Wellman (2003); Grau-Moya et al. (2018) extends the Q-learning algorithm (Watkins and Dayan , 1992) to zero-sum and general-sum Markov games, and that in Perolat et al. (2018); Srinivasan…”
Cooperative Multi-Agent Learning: The State of the Art (2005, background): “Littman [148] introduces the Friend-or-Foe Q-learning algorithm to find either a coordination equilibrium (with cooperative agents) or an adversarial one (with competitive agents).”
Influencing Long-Term Behavior in Multiagent Reinforcement Learning (2022, unknown): “Another effective approach to addressing the non-stationarity is learning equilibrium policies that correspond to game-theoretic solution concepts (Littman, 1994; 2001; Wang & Sandholm, 2002; Greenwald & Hall, 2003; Zinkevich et al., 2006).”
Provable Self-Play Algorithms for Competitive Reinforcement Learning (2020, background): “In this paper, we consider zero-sum Markov Games (MG) (Shapley, 1953; Littman, 1994), which also known as stochastic games in the literature.”
An Overview of Cooperative and Competitive Multiagent Learning (2005, background): “Concurrent learning algorithms for such settings 3 range from Nash-Q [76], Friend-or-Foe Q-learning [93], EXORL ([161]), Correlated-Q [62], to WolF [27] .”