Fictitious Self-Play in Extensive-Form Games

By Johannes Heinrich, Marc Lanctot, David Silver. 2015

346 citations recorded by the source corpus.

Subfields: Reinforcement Learning, Game & Decision Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Game & Decision Theory

Citations in the corpus

OpenSpiel: A Framework for Reinforcement Learning in Games (2019, methodology): “The next example demonstrates computing α -Rank on an asymmetric 3-player meta-game, constructed by computing utilities for Kuhn poker agents from the best response policies generated in the first few rounds of via extensive-form fictitious play (XFP) [25].”
Deep Reinforcement Learning from Self-Play in Imperfect-Information Games (2016, methodology): “Heinrich et al. (2015) introduce Full-Width Extensive-Form Fictitious Play (XFP) that enables ﬁctitious players to update their strategies in behavioural, extensive form, resulting in linear time and space complexity.”
A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning (2017, background): “We now compare to Neural Fictitious Self-Play (NFSP) [40], an implementation of ﬁctitious play in sequential games using reinforcement learning.”
A survey and critique of multiagent deep reinforcement learning (2018, methodology): “bining game theory and MAL [2, 6, 193]. From that context, some approaches were inspired by in uential game theory approaches. Neural Fictitious Self-Play (NFSP) [173] builds on ctitious (self-) play [190, 249], together with two deep networks to nd approximate Nash equilibria14 in two-player imperf”
Open-ended Learning in Symmetric Zero-sum Games (2019, methodology): “A closely related algorithm is fictitious (self-)play (Brown, 1951; Leslie & Collins, 2006; Heinrich et al., 2015).”
Arena: A General Evaluation Platform and Building Toolkit for Multi-Agent Intelligence (2019, background): “Fictitious Self-Play [33, 34, 35] ﬁrst shows promising performance on the competitive game Leduc Poker.”
Interactive Agent Modeling by Learning to Probe (2018, methodology): “For instance, inspired by game theory, there have been approaches aiming at finding Nash equilibira in multi-agent games, where agents’ models are represented by their utilities (Littman, 1994; Hu & Wellman, 2003) and strategies (Claus & Boutilier, 1998; Tesauro, 2004; Powers & Shoham, 2005; Heinric”
From Poincaré Recurrence to Convergence in Imperfect Information Games: Finding Equilibrium via Regularization (2020, methodology): “, 2009) (which suffer from the aforementioned time-averaging problem), best response algorithms (Heinrich et al., 2015; Lanctot et al., 2017; Heinrich & Silver, 2016), model free reinforcement learning methods (Srinivasan et al.”
Computing Approximate Equilibria in Sequential Adversarial Games by Exploitability Descent (2019, methodology): “So, unlike XFP and CFR, there is no need to compute the average policy.”
Actor-Critic Policy Optimization in Partially Observable Multiagent Environments (2018, background): “However, computing the average policy is complex [31, 102] and potentially worse with function approximation, requiring storing past data in large buffers [32].”
Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms (2019, background): “rium of a two-player zero-sum normal form game. Furthermore, ﬁctitious play methods have also been extended to RL settings without the model knowledge. Speciﬁcally, using sequence-form representation,Heinrich et al. (2015) has proposed the ﬁrst ﬁctitious play algorithm for extensive-form games which”
Human-Timescale Adaptation in an Open-Ended Task Space (2023, methodology): “Co-players for our training tasks are generated using ﬁctitious self-play (Heinrich et al., 2015) and then curated using PLR, as in Samvelyan et al. (2022).”
Deep Fictitious Play for Games with Continuous Action Spaces (2019, methodology): “Existing methods which employ stochastic policies do it in domains with discrete action sets since explicit distributions can be maintained over them [4, 8, 9, 17].”
α-Rank: Multi-Agent Evaluation by Evolution (2019, methodology): “es used for the three and four player variants can be found in [68, Section 4.1]. Here, our meta-game dataset consists of several (ﬁxed) rounds of extensive-form ﬁctitious play (speciﬁcally, XFP from [69]): in round 0, the payoff corresponding to strategy proﬁle (0;0;0)in each meta-game of 3-player ”