Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learning

By Ying Wen, Yaodong Yang, Rui Luo, Jun Wang, Wei Pan. 2019

169 citations recorded by the source corpus.

Humans are capable of attributing latent mental contents such as beliefs, or intentions to others. The social skill is critical in everyday life to reason about the potential consequences of their behaviors so as to plan ahead. It is known that humans use this reasoning ability recursively, i.e. considering what others believe about their own beliefs. In this paper, we start from level-1 recursion and introduce a probabilistic recursive reasoning (PR2) framework for multi-agent reinforcement learning. Our hypothesis is that it is beneficial for each agent to account for how the opponents would react to its future behaviors. Under the PR2 framework, we adopt variational Bayes methods to approximate the opponents' conditional policy, to which each agent finds the best response and then improve their own policy. We develop decentralized-training-decentralized-execution algorithms, PR2-Q and PR2-Actor-Critic, that are proved to converge in the self-play scenario when there is one Nash equilibrium. Our methods are tested on both the matrix game and the differential game, which have a non-trivial equilibrium where common gradient-based methods fail to converge. Our experiments show that it is critical to reason about how the opponents believe about what the agent believes. We expect our work to contribute a new idea of modeling the opponents to the multi-agent reinforcement learning community.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

Celebrating Diversity in Shared Multi-Agent Reinforcement Learning (2021, background): “COMA [20], MADDPG [9], PR2 [27], and DOP [10] study the problem of policy-based multi-agent reinforcement learning.”
RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, background): “PR2 (Wen et al., 2019) and MAAC (Iqbal & Sha, 2019) extend the CCDA paradigm by introducing the mechanism of recursive reasoning and attention, respectively.”
Learning Nearly Decomposable Value Functions Via Communication Minimization (2019, background): “COMA (Foerster et al., 2018), MADDPG (Lowe et al., 2017), and PR2 (Wen et al., 2019) explores multi-agent policy gradients and respectively address the problem of credit assignment, learning in mixed environments and recursive reasoning.”
ROMA: Multi-Agent Reinforcement Learning with Emergent Roles (2020, neutral): “COMA (Foerster et al., 2018), MADDPG (Lowe et al., 2017), PR2 (Wen et al., 2019), and MAAC (Iqbal & Sha, 2019) explore multi-agent policy gradients.”
Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning (2021, methodology): “Consequently, fruitful multi-agent policy gradient algorithms have been developed (Foerster et al., 2018; Peng et al., 2017; Zhang et al., 2020; Wen et al., 2018; 2020; Yang et al., 2018).”
Off-Policy Multi-Agent Decomposed Policy Gradients (2020, methodology): “Several subsequent works make improvements to the CCDA framework by introducing the mechanism of recursive reasoning (Wen et al., 2019) or attention (Iqbal & Sha, 2019).”
Multi-Agent Reinforcement Learning is a Sequence Modeling Problem (2022, methodology): “PR2 [42] and GR2 [43] methods conduct recursive reasoning under the CTDE framework.”
Learning Hierarchical Teaching Policies for Cooperative Agents (2019, background): “COBP and CTBP are comparable to other recent MARL scenarios [20, 39] but are more difficult because of the inherent delayed rewards associated with the fact that the box moves iff both agents learn to push it together.”
Influencing Long-Term Behavior in Multiagent Reinforcement Learning (2022, unknown): “Other related works are the opponent modeling frameworks that infer opponents’ policies and condition an agent’s policy on the inferred information about others (He et al., 2016; Raileanu et al., 2018; Grover et al., 2018; Wen et al., 2019).”
A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learning (2020, background): “Studies regarding the centralized training with decentralized execution framework (Lowe et al., 2017; Foerster et al., 2018b; Yang et al., 2018; Wen et al., 2019), which accounts for the behavior of others through a centralized critic, can also be classified into this category.”
Learning to Communicate Implicitly by Actions (2018, result): “In contrast to works such as IPOMDP (Gmytrasiewicz and Doshi 2005) and PR2 (Wen et al. 2019) where the nested belief is embedded into the training agent’s opponent model, we incorporate level-1 nested belief “I believe that you believe” into our policy by a novel auxiliary reward.”
Influence-Based Multi-Agent Exploration (2019, background): “Although multi-agent reinforcement learning (MARL) has been making significant progresses in recent years (Foerster et al., 2018; Lowe et al., 2017; Wen et al., 2019; Iqbal & Sha, 2019a; Sunehag et al., 2018; Son et al., 2019; Rashid et al., 2018), less attention has been drawn to multi-agent explor”
Theory of Mind and Preference Learning at the Interface of Cognitive Science, Neuroscience, and AI: A Review (2022, background): “Some examples of recent proposals of recursive reasoning models, relevant for higher-order ToM (in a Reinforcement Learning framework), include Wen et al. (2019) and Moreno et al. (2021).”
Consequentialist conditional cooperation in social dilemmas with imperfect information (2017, background): “…A recent stream of research has taken a particular interest in social dilemmas, situations where individuals have incentives to act in ways that undermine socially optimal outcomes (Leibo et al. 2017), (Perolat et al. 2017), (Lerer and Peysakhovich 2017), (Kleiman-Weiner et al. 2016).”