Playing repeated games with large language models
By Elif Akata, L. Schulz, Julian Coda-Forno, Seong Joon Oh, Matthias Bethge, Eric Schulz. Nature Human Behaviour, 2023
295 citations recorded by the source corpus.
Large language models (LLMs) are increasingly used in applications where they interact with humans and other agents. We propose to use behavioural game theory to study LLMs’ cooperation and coordination behaviour. Here we let different LLMs play finitely repeated 2 × 2 games with each other, with human-like strategies, and actual human players. Our results show that LLMs perform particularly well at self-interested games such as the iterated Prisoner’s Dilemma family. However, they behave suboptimally in games that require coordination, such as the Battle of the Sexes. We verify that these behavioural signatures are stable across robustness checks. We also show how GPT-4’s behaviour can be modulated by providing additional information about its opponent and by using a ‘social chain-of-thought’ strategy. This also leads to better scores and more successful coordination when interacting with human players. These results enrich our understanding of LLMs’ social behaviour and pave the way for a behavioural game theory for machines. Large language models perform well in self-interested games such as the iterated Prisoner’s Dilemma but struggle in games that require coordination. Social reasoning strategies can improve cooperative outcomes with both other models and human players.
Subfields: Artificial Intelligence, Cultural Studies, Sociology and Political Science
View Paper: https://www.semanticscholar.org/paper/3f98cf521222c65522200037c0eb95a17081b2dd
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Nitesh V. Chawla
- Zili Wang
- Chenglin Wu
- Xiawu Zheng
- Francisco Herrera
- Gianluca Bontempi
- Andrea Dal Pozzolo
- Charu C. Aggarwal
In Sociology and Political Science
Citations in the corpus
- A survey on large language model based autonomous agents (2023, unknown): “Psychology: LLM-based agents can be used in Psychology for conducting psychological experiments [1, 3, 95, 163].”
- MetaGPT: Meta Programming for Multi-Agent Collaborative Framework (2023, background): “There have been many previous explorations of multi-agent, including collective thinking [4, 5, 6, 7], conversation dataset collection [2, 37], sociological phenomenon research [1, 38], collaboration for efficiency [3, 8, 2, 10].”
- Large Language Model based Multi-Agents: A Survey of Progress and Challenges (2024, background): “…different game settings for exploring or testing game theory hypotheses including reasoning, cooperation, persuasion, deception, leadership, etc. [Akata et al. , 2023] leverages behavioral game theory to examine LLMs’ behavior in interactive social settings, particularly their performance in…”