Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies

By Gati Aher, RosaI. Arriaga, Adam Tauman Kalai. International Conference on Machine Learning, 2022

755 citations recorded by the source corpus.

We introduce a new type of test, called a Turing Experiment (TE), for evaluating to what extent a given language model, such as GPT models, can simulate different aspects of human behavior. A TE can also reveal consistent distortions in a language model's simulation of a specific human behavior. Unlike the Turing Test, which involves simulating a single arbitrary individual, a TE requires simulating a representative sample of participants in human subject research. We carry out TEs that attempt to replicate well-established findings from prior studies. We design a methodology for simulating TEs and illustrate its use to compare how well different language models are able to reproduce classic economic, psycholinguistic, and social psychology experiments: Ultimatum Game, Garden Path Sentences, Milgram Shock Experiment, and Wisdom of Crowds. In the first three TEs, the existing findings were replicated using recent models, while the last TE reveals a"hyper-accuracy distortion"present in some language models (including ChatGPT and GPT-4), which could affect downstream applications in education and the arts.

Subfields: Artificial Intelligence, Sociology and Political Science

View Paper: https://www.semanticscholar.org/paper/59b3fbf146b29b581b677ec4384f14cee87997a4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Sociology and Political Science

Nitesh V. Chawla

Citations in the corpus

The rise and potential of large language model based agents: a survey (2025, neutral): “This metric primarily encompasses whether the agent achieves stipulated objectives or attains expected scores [90, 365,485].”
PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits (2023, background): “…abilities as agents (Andreas, 2022) and exhibit human-like behavior in reasoning (Dasgupta et al., 2022; Webb et al., 2023; Binz and Schulz, 2023; Aher et al., 2023; Wong et al., 2023), role-playing (Wang et al., 2023b; Shao et al., 2023; Wang et al., 2023a), and social science experiments…”
A survey on large language model based autonomous agents (2023, neutral): “For a comprehensive overview, please refer to the right portion of Figure 5.”
Large language models empowered agent-based modeling and simulation: a survey and perspectives (2023, neutral): “More importantly, recent works show the simulations of multiple LLM agents show human-like collective behaviors (Aher et al., 2023; Zhou et al., 2024b), such as social conformity and homophily, which could be exploited by adversaries as weaknesses in the societies of LLM agents.”
S3: Social-network Simulation System with Large Language Model-Empowered Agents (2023, neutral): “[1] conducted a preliminary test to find that LLMs possess the capability to reproduce some classic economic, psycholinguistic, and social psychology experiments.”
Playing repeated games with large language models (2023, neutral): “For example, 40 let GPT-3 participate in the dictator game, and 41 used the same approach for the ultimatum game.”
Large Language Model based Multi-Agents: A Survey of Progress and Challenges (2024, background): “[ Aher et al. , 2023 ] introduces Turing Experiments to evaluate the extent to which large language models can simulate different aspects of human behaviors.”
Multi-Agent Collaboration Mechanisms: A Survey of LLMs (2025, neutral): “Authors in [5, 36] suggest that LLM-based agents can replace human participants in specific social science experiments, while [88] demonstrates their use in generating diverse natural language data with iterative cross-agent refinement.”
DeepInception: Hypnotize Large Language Model to Be Jailbreaker (2023, background): “Besides, a recent work [1] reveals that LLMs behave consistently with the prior human study, where the emergent abilities of instruction following and step-by-step reasoning contribute significantly [20, 24].”
Gaussian mixture models as a proxy for interacting language models (2025, unknown): “Research and development of large language models (LLMs) has progressed considerably in the past several years, with the result that large language models can now replicate several human characteristics and interactions observed in linguisitic, cognitive and social sciences [1, 2].”
Toward a digital twin of U.S. Congress (2025, unknown): “For example, GPT-4 has demonstrated human-like behavior in classical economic, psycholinguistic, and social psychology experiments such as The Ultimatum Game, Garden Path Sentences, Milgram Shock Experiment, and Wisdom of Crowds (3).”
Investigating social alignment via mirroring in a system of interacting language models (2024, unknown): “For example, the recent cohort of Large Language Models (LLMs), such as OpenAI’s GPT4, are able to replicate the Ultimatum Game, Garden Path Sentences, and Milgram Shock Experiment (Aher et al., 2023).”