Markov Games as a Framework for Multi-Agent Reinforcement Learning

By Michael L. Littman. 1994

Subfields: Reinforcement Learning, Game & Decision Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Game & Decision Theory

Citations in the corpus

OpenSpiel: A Framework for Reinforcement Learning in Games (2019, methodology): “Games in OpenSpiel are represented as procedural extensive-form games [52, 63], though in some cases can also be cyclic such as in Markov Decision Processes [67] and Markov games [37].”
The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games (2021, background): “In decentralized learning [20], each agent optimizes its reward independently; these methods can tackle general-sum games but may suffer from instability even in simple matrix games [12].”
Value-Decomposition Networks For Cooperative Multi-Agent Learning (2017, background): “As is standard in MARL, the underlying environment is modeled as a Markov game where actions are chosen and executed simultaneously, and new observations are perceived simultaneously as a result of a transition to a new state (Littman, 1994, 2001; Hu and Wellman, 2003; Busoniu et al., 2008).”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Multi-agent environments are often characterized as a Markov game (Littman, 1994) in which a set of agents all interact in the environment and jointly impact the reward and transition dynamics.”
Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments (2017, methodology): “Markov Games In this work, we consider a multi-agent extension of Markov decision processes (MDPs) called partially observable Markov games [19].”
Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games (2017, methodology): “The studies on interaction and collaboration in multiagent settings have a long history (Littman 1994; Schmidhuber 1996).”
Emergence of Grounded Compositional Language in Multi-Agent Populations (2017, methodology): “The recent work that is most similar to ours is the application of reinforcement learning approaches towards the purposes of learning a communication protocol, as exempliﬁed by (Bratman et al. 2010; Foerster et al. 2016; Sukhbaatar, Szlam, and Fergus 2016; Lazaridou, Peysakhovich, and Ba-roni 2016).”
Multi-agent Reinforcement Learning in Sequential Social Dilemmas (2017, background): “Most previous work on finding policies for Markov games takes the prescriptive view of multiagent learning [26]: that is, it attempts to answer “what should each agent do?” Several algorithms and analyses have been developed for the two-player zero-sum case [22, 27, 28, 29, 30].”
Episodic Exploration for Deep Deterministic Policies: An Application to StarCraft Micromanagement Tasks (2016, background): “Most of the focus has been on learning agents in competitive environments with adaptive adversaries (e.g., [15, 13, 40]).”
Empirically Evaluating Multiagent Learning Algorithms (2014, methodology): “We thus did not make use of work that insists (e.g.) on two-action games (Singh et al., 2000) or constant-sum games (Littman, 1994).”
Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharing (2021, background): “Markov Games: A Markov game (e.g. (Littman, 1994); also known as a stochastic game (Shapley, 1953)) with partial observability is deﬁned by the tuple ( N , S , { O i } i ∈N , { A i } i ∈N , P , { R i } i ∈N ) , with agents i ∈ N = { 1 , . . . , N } , state space S , joint observation space O = O 1…”
The Hanabi Challenge: A New Frontier for AI Research (2019, background): “The setting with multiple reinforcement learning agents was first investigated in competitive games [57].”
Learning to Communicate to Solve Riddles with Deep Distributed Recurrent Q-Networks (2016, background): “We ﬁrst evaluate DDRQN for n = 10 and compare it with tabular Q-learning.”
Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery (2019, methodology): “archical MARL in discrete state spaces with hand-crafted subtasks [12, 22] showed that learning cooperation at the level of subtasks significantly speeds up learning over flat methods [20, 35, 39].”
From Few to More: Large-scale Dynamic Multiagent Curriculum Learning (2019, background): “A natural multiagent extension of Markov decision processes (MDPs) are Stochastic Games (SGs) (Littman 1994), which model the dynamic interactions among multiple agents.”
Action Semantics Network: Considering the Effects of Actions in Multiagent Systems (2019, background): “Stochastic games (SGs) (Littman, 1994) are a natural multiagent extension of Markov decision processes (MDPs), which models the dynamic interactions among multiple agents.”
Actor-Attention-Critic for Multi-Agent Reinforcement Learning (2018, methodology): “et al., 2010). Topics within MARL are diverse, ranging from learning communication between cooperative agents (Tan, 1993; Fischer et al., 2004) to algorithms for optimal play in competitive settings (Littman, 1994), though, until recently, they have been focused on simple gridworld environments with”
Factorized Q-learning for large-scale multi-agent systems (2018, methodology): “To address the difficult decision problems arising from MARL, researchers have tried to borrow techniques from game theory, in particular, the framework of stochastic games [8].”
Mean Field Multi-Agent Reinforcement Learning (2018, methodology): “Our work follows the same direction as Littman (1994); Hu & Wellman (2003); Bowling & Veloso (2002) on adapting a Stochastic Game (van der Wal et al., 1981) into the MARL formulation.”
Multiagent Learning: Basics, Challenges, and Prospects (2012, methodology): “The first multiagent reinforcement learning efforts appeared soon with the work of Whitehead, Tan, and Littman (Whitehead 1991, Tan 1993, Littman 1994).”