Evaluating Large Language Models in Generating Synthetic HCI Research Data: a Case Study

By Perttu Hämäläinen, Mikke Tavast, Anton Kunnari. International Conference on Human Factors in Computing Systems, 2023

330 citations recorded by the source corpus.

Collecting data is one of the bottlenecks of Human-Computer Interaction (HCI) research. Motivated by this, we explore the potential of large language models (LLMs) in generating synthetic user research data. We use OpenAI’s GPT-3 model to generate open-ended questionnaire responses about experiencing video games as art, a topic not tractable with traditional computational user models. We test whether synthetic responses can be distinguished from real responses, analyze errors of synthetic data, and investigate content similarities between synthetic and real data. We conclude that GPT-3 can, in this context, yield believable accounts of HCI experiences. Given the low cost and high speed of LLM data generation, synthetic data should be useful in ideating and piloting new experiments, although any findings must obviously always be validated with real data. The results also raise concerns: if employed by malicious users of crowdsourcing services, LLMs may make crowdsourcing of self-report data fundamentally unreliable.

Subfields: Computer Science Applications, Information Systems, Sociology and Political Science

View Paper: https://www.semanticscholar.org/paper/0ffd57884d7957f6b5634b9fa24843dc3759668f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Information Systems

In Sociology and Political Science

Citations in the corpus

Humans or LLMs as the Judge? A Study on Judgement Bias (2024, methodology): “In Section 6, we provide a simple yet effective prompt-based attack on LLM-as-a-judge.”
Generative Agents: Interactive Simulacra of Human Behavior (2023, background): “Other empirical studies have replicated existing social science studies [46], political surveys [92], and generated synthetic data [39].”
GPT (Generative Pre-Trained Transformer)— A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directions (2023, background): “This makes it easy for the users to interact with GPT models and understand their responses [72].”
A Survey on Large Language Models for Code Generation (2024, unknown): “between smaller and larger LLMs. 7B/13B models outperform the 70B model.”
The Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practice (2023, background): “Meanwhile, some researchers have begun to leverage language models to generate synthetic user research data positing such data as suitable “proxies for understanding human experiences” [10, 91].”
S3: Social-network Simulation System with Large Language Model-Empowered Agents (2023, unknown): “Recently, Part et al. [32] construct a virtual town with 25 LLM-empowered agents based on a video game environment, in which the agent can plan and schedule what to do in daily life.”
LLM-Based Doppelgänger Models: Leveraging Synthetic Data for Human-Like Responses in Survey Simulations (2024, unknown): “An experiment using GPT-3 [1] generates open-ended survey responses about experiencing video games as art and compares them to actual responses, concluding that validation with real data is necessary [7].”