PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits
By Hang Jiang, Xiajie Zhang, Xubo Cao, Cynthia Breazeal, Deb Roy, Jad Kabbara. NAACL-HLT, 2023
301 citations recorded by the source corpus.
Despite the many use cases for large language models (LLMs) in creating personalized chatbots, there has been limited research on evaluating the extent to which the behaviors of personalized LLMs accurately and consistently reflect specific personality traits. We consider studying the behavior of LLM-based agents which we refer to as LLM personas and present a case study with GPT-3.5 and GPT-4 to investigate whether LLMs can generate content that aligns with their assigned personality profiles. To this end, we simulate distinct LLM personas based on the Big Five personality model, have them complete the 44-item Big Five Inventory (BFI) personality test and a story writing task, and then assess their essays with automatic and human evaluations. Results show that LLM personas' self-reported BFI scores are consistent with their designated personality types, with large effect sizes observed across five traits. Additionally, LLM personas' writings have emerging representative linguistic patterns for personality traits when compared with a human writing corpus. Furthermore, human evaluation shows that humans can perceive some personality traits with an accuracy of up to 80%. Interestingly, the accuracy drops significantly when the annotators were informed of AI authorship.
Subfields: Applied Psychology, Human-Computer Interaction
View Paper: https://www.semanticscholar.org/paper/f81d85025b349694a57607c5b0c58e0be70bf22f
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Applied Psychology
In Human-Computer Interaction
Citations in the corpus
- SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents (2023, methodology): “Future versions of SOTOPIA-EVAL may further improve LLM-based evaluation quality using recent methods, such as involving multiple LLMs Chan et al. (2023) and training larger LLM evaluators Zhang et al. (2023).”
- Supernova Event Dataset: Interpreting Large Language Models' Personality through Critical Event Analysis (2025, unknown): “…traits when prompted to assume specific roles, and these traits are often recognizable to human evaluators based on the generated content (Jiang et al., 2023 Although most existing work focuses on improving the accuracy of event extraction from long-form text, our work introduces a new,…”