Actor-Critic Policy Optimization in Partially Observable Multiagent Environments

By Sriram Srinivasan, Marc Lanctot, Vinícius Flores Zambaldi, Julien Pérolat, Karl Tuyls, R´emi Munos, Michael H. Bowling. 2018

159 citations recorded by the source corpus.

Optimization of parameterized policies for reinforcement learning (RL) is an important and challenging problem in artificial intelligence. Among the most common approaches are algorithms based on gradient ascent of a score function representing discounted return. In this paper, we examine the role of these policy gradient and actor-critic algorithms in partially-observable multiagent environments. We show several candidate policy update rules and relate them to a foundation of regret minimization and multiagent learning techniques for the one-shot and tabular cases, leading to previously unknown convergence guarantees. We apply our method to model-free multiagent reinforcement learning in adversarial sequential decision problems (zero-sum imperfect information games), using RL-style function approximation. We evaluate on commonly used benchmark Poker domains, showing performance against fixed policies and empirical convergence to approximate Nash equilibria in self-play with rates similar to or better than a baseline model-free algorithm for zero-sum games, without any domain-specific state space reductions.

Subfields: Artificial Intelligence, Economics and Econometrics, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Economics and Econometrics

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

A survey and critique of multiagent deep reinforcement learning (2018, background): “This lead to new convergence properties of independent RL algorithms in zero-sum games with imperfect information [300].”
From Poincaré Recurrence to Convergence in Imperfect Information Games: Finding Equilibrium via Regularization (2020, methodology): “Our work sheds a new light on those results as the convergence may have been the result of high regularization (the entropy cost added in (Srinivasan et al., 2018) appendix G was 0.1).”
A Theoretical Analysis of Deep Q-Learning (2019, background): “See, e.g., Bowling (2001); Conitzer and Sandholm (2007); Prasad et al. (2015); Wei et al. (2017); Pérolat et al. (2018); Srinivasan et al. (2018); Wei et al. (2017) and the references therein.”
Deep Counterfactual Regret Minimization (2018, unknown): “Model-free policy gradient algorithms have been shown to minimize regret when parameters are tuned appropriately (Srinivasan et al., 2018) and achieve performance comparable to NFSP.”
CM3: Cooperative Multi-goal Multi-stage Multi-agent Reinforcement Learning (2018, background): “While the framework of multi-agent reinforcement learning (MARL) [40, 17, 35, 9, 30] has been equipped with successful methods in deep reinforcement learning (RL) [20, 15] and shown promise on high-dimensional problems with complex agent interactions [19, 21, 13, 6, 16, 34], learning multi-agent coo”
Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms (2019, background): “It has been shown in Srinivasan et al. (2018) that the Qi CF in (4.”
Learning Zero-Sum Simultaneous-Move Markov Games Using Function Approximation and Correlated Equilibrium (2020, background): “…Greenwald et al. (2003); Hu and Wellman (2003); Grau-Moya et al. (2018) extends the Q-learning algorithm (Watkins and Dayan , 1992) to zero-sum and general-sum Markov games, and that in Perolat et al. (2018); Srinivasan et al. (2018) extends the actor-critic algorithm (Konda and Tsitsiklis, 2000).”
Computing Approximate Equilibria in Sequential Adversarial Games by Exploitability Descent (2019, methodology): “With step sizes α t = t − 12 , 0 < t ≤ T , each local GIGA instance has regret after T iterations upper bounded by where et al. , 2018, Lemma 5].”
Rethinking Formal Models of Partially Observable Multiagent Decision Making (2019, methodology): “We would particularly like to highlight the paper [46], which ﬁrst expressed a number of the CFR-related concepts in the notation traditionally used in MARL.”
Multi-objective multi-agent decision making: a utility-based analysis and survey (2019, background): “Recent advances in single and multi-agent RL make use of deep ANNs as function approximators; this emerging paradigm is known as deep reinforcement learning (DRL).”
Neural Replicator Dynamics (2019, methodology): “Under perfect recall, the distribution of the states can be obtained only from the opponents’ policies using Bayes’ rule (see Srinivasan et al. [57, Section 3.2]).”
Distributed off-Policy Actor-Critic Reinforcement Learning with Policy Consensus (2019, methodology): “hallenge in applying the AC method in this scenario is that the Critic update needs the local reward information from all the agents. This usually requires a master node to serve as a central critic, [10,11]. When the network size is large, having a master node introduces signiﬁcant communication ov”
Multi-agent deep reinforcement learning: a survey (2021, background): “In further works, DTDE has been applied to cooperative navigation tasks (Chen et al. 2016; Strouse et al. 2018), to partially observable domains (Dobbe et al. 2017; Nguyen et al. 2017b; Srinivasan et al. 2018), and to social dilemmas (Leibo et al. 2017).”