Out of One, Many: Using Language Models to Simulate Human Samples
By Lisa P. Argyle, E. Busby, Nancy Fulda, Joshua R Gubler, Christopher Rytting, David Wingate. 2022
1,190 citations recorded by the source corpus.
Abstract We propose and explore the possibility that language models can be studied as effective proxies for specific human subpopulations in social science research. Practical and research applications of artificial intelligence tools have sometimes been limited by problematic biases (such as racism or sexism), which are often treated as uniform properties of the models. We show that the “algorithmic bias” within one such tool—the GPT-3 language model—is instead both fine-grained and demographically correlated, meaning that proper conditioning will cause it to accurately emulate response distributions from a wide variety of human subgroups. We term this property algorithmic fidelity and explore its extent in GPT-3. We create “silicon samples” by conditioning the model on thousands of sociodemographic backstories from real human participants in multiple large surveys conducted in the United States. We then compare the silicon and human samples to demonstrate that the information contained in GPT-3 goes far beyond surface similarity. It is nuanced, multifaceted, and reflects the complex interplay between ideas, attitudes, and sociocultural context that characterize human attitudes. We suggest that language models with sufficient algorithmic fidelity thus constitute a novel and powerful tool to advance understanding of humans and society across a variety of disciplines.
Subfields: Communication, General Social Sciences, Political Science and International Relations, Large Language Models, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Communication
- Meredith Ringel Morris
- Yulia Tsvetkov
- Michael S. Bernstein
- Diyi Yang
- F. Delgado
- Stephen Yang
- Qian Yang
- Michael Madaio
In General Social Sciences
- RosaI. Arriaga
- Tianyi Zhang
- Yulia Tsvetkov
- Percy Liang
- Michael S. Bernstein
- Ji-rong Wen
- Diyi Yang
- Tatsunori B. Hashimoto
In Political Science and International Relations
- Meredith Ringel Morris
- Yulia Tsvetkov
- Percy Liang
- Adam Tauman Kalai
- Patrick Y. Wu
- Jonathan Nagler
- Esin Durmus
- Fabrizio Gilardi
In Large Language Models
- RosaI. Arriaga
- Tianyi Zhang
- Omar Shaikh
- Yulia Tsvetkov
- Percy Liang
- Ji-rong Wen
- Diyi Yang
- Pamela Mishkin
In Evaluation & Benchmarks
- Percy Liang
- Esin Durmus
- Tatsunori B. Hashimoto
- Faisal Ladhak
- Yulia Tsvetkov
- Tianyi Zhang
- Carlos Guestrin
- Diyi Yang
Citations in the corpus
- ChatGPT outperforms crowd workers for text-annotation tasks (2023, background): “…very well for a wide range of purposes, including ideological scaling (Wu et al., 2023), the classification of legislative proposals (Nay, 2023), the resolution of cognitive psychology tasks, (Binz and Schulz, 2023), and the simulation of human samples for survey research (Argyle et al., 2023).”
- GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models (2023, background): “1This is distinct from recent social science research that makes use of LLMs to simulate human behavior (Horton, 2023; Sorensen et al., 2022) 2While our exposure rubric does not necessarily tie the concept of language models to any particular model, we were strongly motivated by our observed capabil”
- Evaluating Large Language Models in Generating Synthetic HCI Research Data: a Case Study (2023, background): “…known to make factual errors, but can generate text in many literary styles [8], perform sentiment analysis [64], generate human-like self-reports of emotion [76], and predict how political views afect voting behavior and which words people associate with members of diferent political parties [2].”
- Generative Agents: Interactive Simulacra of Human Behavior (2023, background): “Other empirical studies have replicated existing social science studies [46], political surveys [92], and generated synthetic data [39].”
- Can Large Language Models Transform Computational Social Science? (2023, background): “, 2023), and subjective experience description (Argyle et al., 2022).”
- AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback (2023, background): “Our goal of emulating human annotators also connects to work on simulating humans with LMs based on personas [48, 47, 1, 3], as well as works that simulate human behavior in the context of cognitive science, social science, and economics [69, 25].”
- Whose Opinions Do Language Models Reflect? (2023, background): “Another set of studies have examined whether LMs can be used to simulate personas (Park et al., 2022; Argyle et al., 2022; Jiang et al., 2022; Simmons, 2022), akin to our notion of steerability.”
- From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models (2023, background): “While prior works provided analyses of political leanings in LMs (Jiang et al., 2022a; Argyle et al., 2022), they primarily focused on political individuals, rather than the timeless ideological issues grounded in political science literature.”
- Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies (2022, background): “Our work is most similar to concurrent work on simulating human samples from a population by Argyle et al. (2023), which suggests that LMs can effectively represent different subpopulations when prompted with demographic information.”
- A survey on large language model based autonomous agents (2023, methodology): “Social Science Political Science and Economy Out of One [5], Horton [65], Ziems et al.”
- Large Language Models Can Be Used to Estimate the Ideologies of Politicians in a Zero-Shot Learning Setting (2023, background): “Argyle et al. (2023) find that biases in GPT-3 are fine-grained and demographically correlated, and can be used to emulate partisan responses from a wide variety of human subgroups.”
- Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus? (2023, neutral): “Although I do some of this in this paper, Argyle et al. (2022) “endow” LLM agents with demographic characteristics and then get responses in various scenarios that match what is seen empirically.”
- The Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practice (2023, background): “Meanwhile, some researchers have begun to leverage language models to generate synthetic user research data positing such data as suitable “proxies for understanding human experiences” [10, 91].”
- Artificial intelligence and illusions of understanding in scientific research (2024, neutral): “By leveraging the potential of generative AI as an “agent of replacement” 37 , proposals for “silicon subjects” 38 and “synthetic data points” 2 span scientific disciplines.”