Modeling Others using Oneself in Multi-Agent Reinforcement Learning

By Roberta Raileanu, Emily L. Denton, Arthur Szlam, Rob Fergus. 2018

234 citations recorded by the source corpus.

We consider the multi-agent reinforcement learning setting with imperfect information in which each agent is trying to maximize its own utility. The reward function depends on the hidden state (or goal) of both agents, so the agents must infer the other players' hidden goals from their observed behavior in order to solve the tasks. We propose a new approach for learning in these domains: Self Other-Modeling (SOM), in which an agent uses its own policy to predict the other agent's actions and update its belief of their hidden state in an online manner. We evaluate this approach on three different tasks and show that the agents are able to learn better policies using their estimate of the other players' hidden states, in both cooperative and adversarial settings.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks (2020, methodology): “These algorithm classes are frequently used in the literature either as baselines or building blocks for more complex algorithms [He et al., 2016, Sukhbaatar et al., 2016, Foerster et al., 2016, Raileanu et al., 2018, Jaques et al., 2019, Iqbal and Sha, 2019, Du et al., 2019, Ryu et al., 2020].”
A survey and critique of multiagent deep reinforcement learning (2018, methodology): “Self Other Modeling (SOM) [73] proposed a different approach, this is, using the agent’s own policy as a means to predict the opponent’s actions.”
Relational Forward Models for Multi-Agent Learning (2018, neutral): “Despite these recent successes, coordination is still considered a hard problem and several attempts have been made to promote the emergence of coordination by relaxing some assumptions (Foerster et al., 2016; 2017; Sukhbaatar et al., 2016; Raileanu et al., 2018; He et al., 2016a).”
Comparative Evaluation of Multi-Agent Deep Reinforcement Learning Algorithms (2020, background): “These algorithm classes are frequently used in the literature either as baselines or building blocks for more complex algorithms [8, 11, 13, 15, 17, 27, 29, 32].”
Learning to Communicate Implicitly by Actions (2018, background): “Our work is closely related to (Albrecht and Stone 2017; Lowe et al. 2017; Mealing and Shapiro 2017; Raileanu et al. 2018) where agents build models to estimate other agents’ hidden information.”
Towards Open Ad Hoc Teamwork Using Graph-based Policy Learning (2020, background): “Recent agent modeling frameworks explored in deep reinforcement learning (Raileanu et al., 2018; Rabinowitz et al., 2018; He et al., 2016) are designed for closed environments.”
Local Information Agent Modelling in Partially-Observable Environments (2021, background): “Raileanu et al. [2018] developed an algorithm for learning to infer an agent’s intentions using the policy of the controlled agent.”
A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learning (2020, background): “The literature on opponent modeling, for instance, infers opponents’ behaviors and conditions an agent’s policy on the inferred behaviors of others (He et al., 2016; Raileanu et al., 2018; Grover et al., 2018).”
Dealing with Non-Stationarity in MARL via Trust-Region Decomposition (2021, background): “…al., 2019): targeted modiﬁcations on standard RL learning schemes (Lowe et al., 2017; Foerster et al., 2018b; Iqbal & Sha, 2019; Baker et al., 2019), and opponent information estimation and sharing (Raileanu et al., 2018; Foerster et al., 2018a; Rabinowitz et al., 2018; Al-Shedivat et al., 2018).”
Concurrent Meta Reinforcement Learning (2019, unknown): “…have demonstrated impressive performance in the areas of reinforcement learning Finn et al. (2017), learning to explore Gupta et al. (2018b), unsupervised reinforcement learning Gupta et al. (2018a) and multiagent and non-stationary environments Al-Shedivat et al. (2017); Raileanu et al. (2018).”
Multi-agent deep reinforcement learning: a survey (2021, unknown): “Raileanu et al. (2018) employed a model where agents estimate the other agents’ hidden states and embed these estimations into their own policy.”
Robust Opponent Modeling via Adversarial Ensemble Reinforcement Learning in Asymmetric Imperfect-Information Games (2019, background): “In contrast, a more natural approach of opponent modeling is concurrently training all the agents via MARL in a self-play manner [3], [16], [21].”
Influencing Long-Term Behavior in Multiagent Reinforcement Learning (2022, unknown): “Other related works are the opponent modeling frameworks that infer opponents’ policies and condition an agent’s policy on the inferred information about others (He et al., 2016; Raileanu et al., 2018; Grover et al., 2018; Wen et al., 2019).”
ToM2C: Target-oriented Multi-agent Communication and Cooperation with Theory of Mind (2021, methodology): “Motivated by this, machine learning researchers have spent efforts on developing the machine ToM (Rabinowitz et al., 2018) or modeling opponents (He et al., 2016) for multiagent learning (Raileanu et al., 2018; Grover et al., 2018; Lim et al., 2020).”