The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systems

By Caleb Ziems, Jane A. Yu, Yi-Chia Wang, Alon Y. Halevy, Diyi Yang. 2022

133 citations recorded by the source corpus.

Conversational agents have come increasingly closer to human competence in open-domain dialogue settings; however, such models can reflect insensitive, hurtful, or entirely incoherent viewpoints that erode a user’s trust in the moral integrity of the system. Moral deviations are difficult to mitigate because moral judgments are not universal, and there may be multiple competing judgments that apply to a situation simultaneously. In this work, we introduce a new resource, not to authoritatively resolve moral ambiguities, but instead to facilitate systematic understanding of the intuitions, values and moral judgments reflected in the utterances of dialogue systems. The Moral Integrity Corpus, MIC, is such a resource, which captures the moral assumptions of 38k prompt-reply pairs, using 99k distinct Rules of Thumb (RoTs). Each RoT reflects a particular moral conviction that can explain why a chatbot’s reply may appear acceptable or problematic. We further organize RoTs with a set of 9 moral and social attributes and benchmark performance for attribute classification. Most importantly, we show that current neural language models can automatically generate new RoTs that reasonably describe previously unseen interactions, but they still struggle with certain scenarios. Our findings suggest that MIC will be a useful resource for understanding and language models’ implicit moral assumptions and flexibly benchmarking the integrity of conversational agents. To download the data, see https://github.com/GT-SALT/mic

Subfields: Artificial Intelligence, Health Informatics, Natural Language Processing, Alignment & RLHF

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Alignment & RLHF

Yejin Choi

Citations in the corpus

Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, background): “Many works let models cooperate with human annotators(Kim et al., 2022; Ziems et al., 2022; Hartvigsen et al., 2022; Baheti et al., 2021; Xu et al., 2020; Ganguli et al., 2022; Deng et al., 2022).”
ProsocialDialog: A Prosocial Backbone for Conversational Agents (2022, methodology): “MIC is a recently released dataset composed of question-answer pairs for benchmarking the morality of the chatbot’s answers, in which human workers annotate RoTs for the chatbot’s responses along with attributes.”
Mirages: On Anthropomorphism in Dialogue Systems (2023, background): “For example, Ziems et al. (2022) present a corpus of conﬂicting human judgements on moral issues which they label according to ‘rules of thumb’ that they hope explain why some system outputs are perceived as acceptable while others are not.”
Towards Identifying Social Bias in Dialog Systems: Framework, Dataset, and Benchmark (2022, unknown): “…2018; Dinan et al., 2021), bias (Henderson et al., 2018; Liu et al., 2020; Barikeri et al., 2021; Lee et al., 2019), privacy (Weidinger et al., 2021), sensitive topics (Xu et al., 2020; Sun et al., 2022), and moral considerations (Ziems et al., 2022; Kim et al., 2022) draw increasing attention.”
Second Thoughts are Best: Learning to Re-Align With Human Values from Text Edits (2023, methodology): “, 2019), or ratings (Ziems et al., 2022; Lourie et al., 2020) to model human value preferences.”