Toxicity in ChatGPT: Analyzing Persona-assigned Language Models

By A. Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, Karthik Narasimhan. 2023

523 citations recorded by the source corpus.

Large language models (LLMs) have shown incredible capabilities and transcended the natural language processing (NLP) community, with adoption throughout many services like healthcare, therapy, education, and customer service. Since users include people with critical information needs like students or patients engaging with chatbots, the safety of these systems is of prime importance. Therefore, a clear understanding of the capabilities and limitations of LLMs is necessary. To this end, we systematically evaluate toxicity in over half a million generations of ChatGPT, a popular dialogue-based LLM. We find that setting the system parameter of ChatGPT by assigning it a persona, say that of the boxer Muhammad Ali, significantly increases the toxicity of generations. Depending on the persona assigned to ChatGPT, its toxicity can increase up to 6x, with outputs engaging in incorrect stereotypes, harmful dialogue, and hurtful opinions. This may be potentially defamatory to the persona and harmful to an unsuspecting user. Furthermore, we find concerning patterns where specific entities (e.g., certain races) are targeted more than others (3x more) irrespective of the assigned persona, that reflect inherent discriminatory biases in the model. We hope that our findings inspire the broader AI community to rethink the efficacy of current safety guardrails and develop better techniques that lead to robust, safe, and trustworthy AI systems.

Subfields: Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

1. Furu Wei
2. Minlie Huang
3. Tao Ge
4. Heng Ji
5. Tatsunori B. Hashimoto
6. Dan Jurafsky
7. Yankai Lin
8. Mohit Bansal

In Safety, Robustness & Interpretability

1. Mantas Mazeika
2. Dan Hendrycks
3. Steven Basart
4. Minlie Huang
5. Andy Zou
6. Alexander Pan
7. Zhexin Zhang
8. Hao Sun

Citations in the corpus

• Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, background): “Levy et al. (2022) 2023) leverage system prompt to assign a role for ChatGPT, like " Speak like Adolf Hitler ", and they find that this can obviously induce the model to be more toxic.”
• Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (2023, background): “When prompted, the model may produce convincing personalized scams given appropriate knowledge about the target [21, 43] (that was either given in the prompt or, importantly , the model acquired during the chat session).”
• The rise and potential of large language model based agents: a survey (2025, unknown): “These phenomena span from individual-level issues like discrimination, isolation, and bullying, to broader concerns such as oppressive slavery and group antagonism [461,462].”
• Large language models empowered agent-based modeling and simulation: a survey and perspectives (2023, unknown): “In the context of LLM-driven agent simulations, researchers fi nd when certain personas are assigned to ChatGPT it will generate output with 6× toxicity, engaging in discriminatory stereotypes, harmful conversation, and offensive language (Deshpande et al., 2023).”
• A Survey on Evaluation of Large Language Models (2023, background): “[33] introduced role-playing into the model and observed an increase in generated toxicity up to 6x.”
• A survey on large language model based autonomous agents (2023, background): “[31] studies the toxicity of the LLM output by manually prompting LLMs with different roles, such as politicians, journalists and businesspersons.”
• BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset (2023, background): “If left unchecked, LLMs can amplify misinformation, enable harmful content, or yield unintended responses that can cause significant negative societal impact [16, 17, 18, 19].”
• Jailbreaking Black Box Large Language Models in Twenty Queries (2023, unknown): “However, one drawback of this approach is that these massive corpora often contain toxic or objectionable content, which—when propagated by an LLM trained on this data—has the propensity to cause harm [e.g., 8, etc.].”
• SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks (2023, unknown): “At face value, efforts to align LLMs have reduced the propagation of toxic content: Publicly-available chatbots will now rarely output text that is clearly objectionable [8].”
• The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning (2024, background): “…from LLMs. Recent work has highlighted safety concerns of language models, including generating falsehoods (Ji et al., 2023; Zhang et al., 2023), producing toxic content (Gehman et al., 2020; Deshpande et al., 2023; Pan et al., 2024), and deceiving humans (Park et al., 2023; Scheurer et al., 2023).”
• Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions (2023, background): “There is already an emerging body of scholarly work (Abid et al., 2021; Weidinger et al., 2021; Bommasani et al., 2021; Deshpande et al., 2023; Wei et al., 2023; Zack et al., 2023; Dahl et al., 2024, inter alia ), as well as a plethora of individual accounts on social and traditional media outlets,…”
• Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration (2023, background): “…heavily inspired by the role of pretend play (Piaget, 1954; Pellegrini, 2009) in cognitive development and recent findings that assigning personas (Deshpande et al., 2023; Xu et al., 2023) to LLMs can elicit specific behaviors, improve answer quality, and potentially build an AI society (Park et…”
• GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts (2023, background): “They can produce toxic or mis-leading content [15,21,66] and are susceptible to “hallucina-tions” that result in nonsensical or untruthful outputs [33,45].”
• Safe RLHF: Safe Reinforcement Learning from Human Feedback (2023, unknown): “Deshpande et al. (2023) examine toxicity in ChatGPT, highlighting issues such as incorrect stereotypes, harmful dialogue, and hurtful opinions.”