Autonomous agents modelling other agents: A comprehensive survey and open problems

By Stefano V. Albrecht, Peter Stone. 2017

543 citations recorded by the source corpus.

Abstract Much research in artificial intelligence is concerned with the development of autonomous agents that can interact effectively with other agents. An important aspect of such agents is the ability to reason about the behaviours of other agents, by constructing models which make predictions about various properties of interest (such as actions, goals, beliefs) of the modelled agents. A variety of modelling approaches now exist which vary widely in their methodology and underlying assumptions, catering to the needs of the different sub-communities within which they were developed and reflecting the different practical uses for which they are intended. The purpose of the present article is to provide a comprehensive survey of the salient modelling methods which can be found in the literature. The article concludes with a discussion of open problems which may form the basis for fruitful future research.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharing (2021, methodology): “Recent work has focused on novel techniques for experience sharing (Christianos et al., 2020), agent modelling (Albrecht & Stone, 2018), and communication be-tween agents (Rangwala & Williams, 2020; Zhang et al., 2020) to address the non-stationarity and multi-agent credit assignment problems…”
The Hanabi Challenge: A New Frontier for AI Research (2019, background): “signiﬁcant ﬁnancial support for this work that could have inﬂuenced its outcome.”
Machine Theory of Mind (2018, background): “Our work thus ties in to a rich history of opponent modelling (Brown, 1951; Albrecht & Stone, 2017); within this context, we show how meta-learning could be used to furnish an agent with the ability to build ﬂexible and sample-efﬁcient models of others on the ﬂy.”
Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learning (2019, background): “Modelling other agents in multi-agent systems has been widely studied and offers many research opportunities (Albrecht and Stone, 2018).”
Actor-Critic Policy Optimization in Partially Observable Multiagent Environments (2018, methodology): “There are also methods that attempt to model the opponents [36, 30, 4].”
Algorithmic Pricing and Competition: Empirical Evidence from the German Retail Gasoline Market (2023, methodology): “We look at the effects of adoption on mean monthly margins and prices, and on the distribution of monthly margins and prices (5th, 25th, 50th, 75th and 95th percentiles).(3)”
Local Information Agent Modelling in Partially-Observable Environments (2021, methodology): “Much research has been devoted to this agent modelling problem [Albrecht and Stone, 2018], with recent works focused on learning informative representations about another agent’s policy using deep learning architectures for agent modelling and reinforcement learning (RL) [He et al., 2016, Raileanu…”
Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learning (2019, unknown): “Traditional approach of constructing the models of other agents, also known as opponent modeling, has a rich history in the multi-agent learning (Shoham et al., 2007; Albrecht & Stone, 2018).”
Robust Opponent Modeling via Adversarial Ensemble Reinforcement Learning in Asymmetric Imperfect-Information Games (2019, methodology): “Classic methods, such as policy reconstruction [4], [5] and plan recognition, [7], [25] etc., develop parametric models to model agent behaviors [1].”
Learning to Communicate Implicitly by Actions (2018, background): “Our work is closely related to (Albrecht and Stone 2017; Lowe et al. 2017; Mealing and Shapiro 2017; Raileanu et al. 2018) where agents build models to estimate other agents’ hidden information.”
Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration (2020, unknown): “These fundamental aspects of machine social intelligence have been shown to be key to human-AI collaboration in prior work (Grosz & Kraus, 1996; Albrecht & Stone, 2018).”
Towards Open Ad Hoc Teamwork Using Graph-based Policy Learning (2020, methodology): “Agent modeling: An agent model takes a history of observations (e.g actions, states) as input and produces a prediction about the modeled agent, such as its goals or future actions (Albrecht & Stone, 2018).”
A Deep Policy Inference Q-Network for Multi-Agent Systems (2017, background): “Modeling and exploiting other agents’ behaviors in a multi-agent system (MAS) have received much attention in the past decade [4, 12, 16, 23, 24].”
Theory of Mind and Preference Learning at the Interface of Cognitive Science, Neuroscience, and AI: A Review (2022, background): “…to varying degrees in this review, but detailed reviews for each are provided in the following papers (Game ToM in Yoshida et al., 2008; Observational Reinforcement Learning in Albrecht and Stone, 2018; Inverse Reinforcement Learning in Arora and Doshi, 2021 and Bayesian ToM in Baker et al., 2011).”