Revealing Persona Biases in Dialogue Systems

By Emily Sheng, Josh Arnold, Zhou Yu, Kai-Wei Chang, Nanyun Peng. 2021

46 citations recorded by the source corpus.

Dialogue systems in the form of chatbots and personal assistants are being increasingly integrated into people's lives. Modern dialogue systems may consider adopting anthropomorphic personas, mimicking societal demographic groups to appear more approachable and trustworthy to users. However, the adoption of a persona can result in the adoption of biases. In this paper, we present the first large-scale study on persona biases in dialogue systems and conduct analyses on personas of different social classes, sexual orientations, races, and genders. We define persona biases as harmful differences in responses (e.g., varying levels of offensiveness, agreement with harmful statements) generated from adopting different demographic personas. Furthermore, we introduce an open-source framework, UnitPersonaBias, to explore and aggregate persona biases in dialogue systems. By analyzing the Blender and DialoGPT dialogue systems, we observe that adopting personas can actually decrease harmful responses, compared to not using any personas. Additionally, we find that persona choices can affect the degree of harms in generated responses and thus should be systematically evaluated before deployment. We also analyze how personas can result in different amounts of harm towards specific demographics.

Subfields: Artificial Intelligence, Natural Language Processing, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Human-Computer Interaction

Citations in the corpus

Societal Biases in Language Generation: Progress and Challenges (2021, methodology): “…2019), gender in translation (Prates et al., 2019), names (Shwartz et al., 2020), sentiment distribution (Huang et al., 2020), dialects (Groenwold et al., 2020), dialogue personas (Sheng et al., 2021a), or other notions of similarity across demographics (Yeo and Chen, 2020; Henderson et al., 2018).”
“I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset (2022, methodology): “…style classifier of Smith et al. (2020a) could be to use the uni-axial sentiment classifier VADER (Hutto and Gilbert, 2014), which is used in Sheng et al. (2021a) in part to measure the sentiment of harmful affirmations (i.e. “[DEMOGRAPHIC] are ridiculous”) and in Liu et al. (2020) to…”
Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models (2021, methodology): “…standard training datasets for dialogue models in the AI literature contain social biases or demographic imbalances, and it has been established that models learn them (Dixon et al., 2018; Bordia and Bowman, 2019; Lee et al., 2019; Dinan et al., 2020a,b; Liu et al., 2020a,b; Sheng et al., 2021).”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, methodology): “For dialog systems, Liu et al. (2019) Sheng et al. (2021) study biases relating to the personas assigned to the Blender (Roller et al., 2020) and DialoGPT (Zhang et al., 2019) dialog systems, presenting a tool to measure these biases, and demonstrating that a system’s persona can affect the level…”
Guiding the Release of Safer E2E Conversational AI through Value Sensitive Design (2022, background): “Conditioning generations on certain types of context, such as personas of diverse historically marginalised demographics, has also been shown to decrease the generation of harmful responses (Sheng et al., 2021).”
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage (2022, background): “The second set of tests is the offensive generation test, which is an evaluation of model responses to a constructed set of hateful inputs from Sheng et al. (2021); we measure model affirmation to hateful inputs via three axes: % of responses flagged by a safety classifier, % of responses with no…”
SafetyKit: First Aid for Measuring Safety in Open-domain Conversational Systems (2022, methodology): “Offensive inputs are constructed using the set up described in Sheng et al. (2021): the authors take a list of 17 demographic dimensions along the axes of social class, sexual orientation, race, and gender ( X ) and a list of negative adjectives ( Y ) and construct the input template “ X are Y .””
"I'm sorry to hear that": finding bias in language models with a holistic descriptor dataset (2022, background): “…style classi-ﬁer of Smith et al. (2020a) could be to use the existing uni-axial sentiment classiﬁer VADER (Hutto and Gilbert, 2014), which is used in Sheng et al. (2021a) in part to measure the sentiment of harmful afﬁrmations (i.e. “[DEMOGRAPHIC] are ridicu-lous” ) and in Liu et al. (2020) to…”
Alexa, Google, Siri: What are Your Pronouns? Gender and Anthropomorphism in the Design and Perception of Conversational Assistants (2021, background): “Similarly, Sheng et al. (2021) found that neural chatbots will generate a biased response dependent on which sentence-based persona description was used to initialise the model (following Zhang et al. (2018)).”