Ethical Challenges in Data-Driven Dialogue Systems

By Peter Henderson, Koustuv Sinha, Nicolas Angelard-Gontier, Nan Rosemary Ke, Genevieve Fried, Ryan J. Lowe, Joelle Pineau. 2017

190 citations recorded by the source corpus.

The use of dialogue systems as a medium for human-machine interaction is an increasingly prevalent paradigm. A growing number of dialogue systems use conversation strategies that are learned from large datasets. There are well documented instances where interactions with these system have resulted in biased or even offensive conversations due to the data-driven training process. Here, we highlight potential ethical issues that arise in dialogue systems research, including: implicit biases in data-driven systems, the rise of adversarial examples, potential sources of privacy violations, safety concerns, special considerations for reinforcement learning systems, and reproducibility concerns. We also suggest areas stemming from these issues that deserve further investigation. Through this initial survey, we hope to spur research leading to robust, safe, and ethically sound dialogue systems.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

The Woman Worked as a Babysitter: On Biases in Language Generation (2019, background): “Recent works in machine translation (Prates et al., 2018) and dialogue systems (Henderson et al., 2018) have brought to attention the perpetuation of biases in natural language generation (NLG) systems.”
LaMDA: Language Models for Dialog Applications (2022, background): “For instance, examples of bias, offensiveness, and hate speech have been found both in training data drawn from social media, and consequently in the output of dialog models trained on such data [83].”
Detoxifying Language Models Risks Marginalizing Minority Voices (2021, background): “When trained on such data, LMs inevitably learn to generate toxic text (Henderson et al., 2018; Wallace et al., 2019).”
Quantifying Memorization Across Neural Language Models (2022, background): “, 2020; Ziegler, 2021), or in other work, synthetically injected “canaries” (Carlini et al., 2019; Henderson et al., 2017; Thakkar et al., 2020; Thomas et al., 2020).”
Societal Biases in Language Generation: Progress and Challenges (2021, background): “In terms of recurrent dialogue models, Henderson et al. (2018) analyze biases in hierarchical recurrent encoder-decoder architectures and Liu et al. (2020a,b) analyze LSTM-based encoder-decoder models.”
OpenAssistant Conversations - Democratizing Large Language Model Alignment (2023, background): “LLMs can also produce toxic or hateful content and fail to follow provided constraints [32].”
Pretraining Language Models with Human Preferences (2023, background): “This poses privacy risks if the text contains confidential information identifying living people (PII) such as email addresses or social security numbers (Henderson et al., 2018).”
Recipes for Safety in Open-domain Chatbots (2020, background): “Surveying ethical challenges in dialogue systems, Henderson et al. (2018) note the axes of bias, adversarial examples, privacy, safety, and propose that the community should aim to provide conditional safety guarantees, such as an upper bound on the probability that a model will generate an unsafe…”
Alignment of Language Agents (2021, background): “Henderson et al. (2018) look at dangers with dialogue systems which they take to mean ‘offensive or harmful effects to human interlocutors’.”
Queens Are Powerful Too: Mitigating Gender Bias in Dialogue Generation (2019, background): “Dialogue research has found that, while incorporating personas increases engagingness and improves consistency (Zhang et al., 2018; Shuster et al., 2018; Mazar´e et al., 2018; Olabiyi et al., 2018; Li et al., 2016b), they can also crystallize gender bias (Clark et al., 2019; Henderson et al., 2018).”
Does Gender Matter? Towards Fairness in Dialogue Systems (2019, background): “Offensive responses cause users discomfort and should be avoided (Henderson et al., 2018; Dinan et al., 2019b; Liu et al., 2019; Liu et al., 2020b).”
Red Teaming Language Models with Language Models (2022, background): “Data leakage compromises user privacy when the LM (e.g., GMail autocomplete; Chen et al., 2019) learns from confidential data (e.g., emails with Social Security Numbers; Carlini et al., 2019; Henderson et al., 2018).”
True Few-Shot Learning with Language Models (2021, methodology): “We evaluate the generalization loss of the algorithm chosen by CV (likewise for MDL): where Reliability is important for many practical or commercial applications where worst-case performance is important, such as image recognition [35, 36], dialogue systems [37, 38], and robotics [39, 40].”
Foundation Models and Fair Use (2023, background): “Bommasani et al. (2021), Bender et al. (2021), and Henderson et al. (2018) all provide high level overviews of the potential risks from language model or foundation model deployments.”
Revealing Persona Biases in Dialogue Systems (2021, neutral): “Henderson et al. (2018) is an earlier work that highlights various ethical issues with dialogue systems.”
Counterfactual Memorization in Neural Language Models (2021, background): “…work on memorization in neural language models demonstrated the ability to extract memorized training data, including sensitive data such as phone numbers and usernames [Carlini et al., 2020, Ziegler, 2021, Carlini et al., 2019, Henderson et al., 2017, Thakkar et al., 2020, Thomas et al., 2020].”