Out of One, Many: Using Language Models to Simulate Human Samples

By Lisa P. Argyle, E. Busby, Nancy Fulda, Joshua R Gubler, Christopher Rytting, David Wingate. 2022

1,190 citations recorded by the source corpus.

Abstract We propose and explore the possibility that language models can be studied as effective proxies for specific human subpopulations in social science research. Practical and research applications of artificial intelligence tools have sometimes been limited by problematic biases (such as racism or sexism), which are often treated as uniform properties of the models. We show that the “algorithmic bias” within one such tool—the GPT-3 language model—is instead both fine-grained and demographically correlated, meaning that proper conditioning will cause it to accurately emulate response distributions from a wide variety of human subgroups. We term this property algorithmic fidelity and explore its extent in GPT-3. We create “silicon samples” by conditioning the model on thousands of sociodemographic backstories from real human participants in multiple large surveys conducted in the United States. We then compare the silicon and human samples to demonstrate that the information contained in GPT-3 goes far beyond surface similarity. It is nuanced, multifaceted, and reflects the complex interplay between ideas, attitudes, and sociocultural context that characterize human attitudes. We suggest that language models with sufficient algorithmic fidelity thus constitute a novel and powerful tool to advance understanding of humans and society across a variety of disciplines.

Subfields: Communication, General Social Sciences, Political Science and International Relations, Large Language Models, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Communication

  1. Meredith Ringel Morris
  2. Yulia Tsvetkov
  3. Michael S. Bernstein
  4. Diyi Yang
  5. F. Delgado
  6. Stephen Yang
  7. Qian Yang
  8. Michael Madaio

In General Social Sciences

  1. RosaI. Arriaga
  2. Tianyi Zhang
  3. Yulia Tsvetkov
  4. Percy Liang
  5. Michael S. Bernstein
  6. Ji-rong Wen
  7. Diyi Yang
  8. Tatsunori B. Hashimoto

In Political Science and International Relations

  1. Meredith Ringel Morris
  2. Yulia Tsvetkov
  3. Percy Liang
  4. Adam Tauman Kalai
  5. Patrick Y. Wu
  6. Jonathan Nagler
  7. Esin Durmus
  8. Fabrizio Gilardi

In Large Language Models

  1. RosaI. Arriaga
  2. Tianyi Zhang
  3. Omar Shaikh
  4. Yulia Tsvetkov
  5. Percy Liang
  6. Ji-rong Wen
  7. Diyi Yang
  8. Pamela Mishkin

In Evaluation & Benchmarks

  1. Percy Liang
  2. Esin Durmus
  3. Tatsunori B. Hashimoto
  4. Faisal Ladhak
  5. Yulia Tsvetkov
  6. Tianyi Zhang
  7. Carlos Guestrin
  8. Diyi Yang

Citations in the corpus