Semantics Derived Automatically from Language Corpora Contain Human-like Moral Choices

By Sophie F. Jentzsch, Patrick Schramowski, Constantin A. Rothkopf, Kristian Kersting. 2019

77 citations recorded by the source corpus.

Allowing machines to choose whether to kill humans would be devastating for world peace and security. But how do we equip machines with the ability to learn ethical or even moral choices? Here, we show that applying machine learning to human texts can extract deontological ethical reasoning about "right" and "wrong" conduct. We create a template list of prompts and responses, which include questions, such as "Should I kill people?", "Should I murder people?", etc. with answer templates of "Yes/no, I should (not)." The model's bias score is now the difference between the model's score of the positive response ("Yes, I should'') and that of the negative response ("No, I should not"). For a given choice overall, the model's bias score is the sum of the bias scores for all question/answer templates with that choice. We ran different choices through this analysis using a Universal Sentence Encoder. Our results indicate that text corpora contain recoverable and accurate imprints of our social, ethical and even moral choices. Our method holds promise for extracting, quantifying and comparing sources of moral choices in culture, including technology.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Sociology and Political Science, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Sociology and Political Science

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Alignment of Language Agents (2021, background): “Language agents may give harmful and biased outputs, producing discriminatory content relating to people’s protected characteristics and other sensitive attributes such as someone’s socio-economic status, see e.g. (Jentzsch et al., 2019; Lu et al., 2020; Zhao et al., 2017).”
Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models (2022, unknown): “[21] Sophie Jentzsch, Patrick Schramowski, Constantin A.”
Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content? (2022, background): “Furthermore, Jentzsch et al. [2019] and Schramowski et al. [2020] showed that the retained knowledge of such models carries information about moral norms aligning with the human sense of “right” and “wrong” expressed in language.”
On the Machine Learning of Ethical Judgments from Natural Language (2022, background): “Other work seeks to characterize what language models already implicitly represent about morality by investigating their learned sentence representations (Jentzsch et al., 2019; Schramowski et al., 2020, 2021).”
Scruples: A Corpus of Community Ethical Judgments on 32, 000 Real-Life Anecdotes (2020, background): “Moving from theory-driven to data-driven approaches, other works found that distributional language representations, like word vectors and neural embeddings, encode commonsense notions of normative behavior (Jentzsch et al. 2019; Schramowski et al. 2019).”
Large pre-trained language models contain human-like biases of what is right and wrong to do (2021, methodology): “Furthermore, our proposed approach enables the rating of general sentences or statements, overcoming one of the main limitations of previous approaches [26, 27].”