Large pre-trained language models contain human-like biases of what is right and wrong to do

By Patrick Schramowski, Cigdem Turan, Nico Andersen, Constantin A. Rothkopf, Kristian Kersting. 2021

398 citations recorded by the source corpus.

Artificial writing is permeating our lives due to recent advances in large-scale, transformer-based language models (LMs) such as BERT, GPT-2 and GPT-3. Using them as pre-trained models and fine-tuning them for specific tasks, researchers have extended the state of the art for many natural language processing tasks and shown that they capture not only linguistic knowledge but also retain general knowledge implicitly present in the data. Unfortunately, LMs trained on unfiltered text corpora suffer from degenerated and biased behaviour. While this is well established, we show here that recent LMs also contain human-like biases of what is right and wrong to do, reflecting existing ethical and moral norms of society. We show that these norms can be captured geometrically by a ‘moral direction’ which can be computed, for example, by a PCA, in the embedding space. The computed ‘moral direction’ can rate the normativity (or non-normativity) of arbitrary phrases without explicitly training the LM for this task, reflecting social norms well. We demonstrate that computing the ’moral direction’ can provide a path for attenuating or even preventing toxic degeneration in LMs, showcasing this capability on the RealToxicityPrompts testbed. Large language models identify patterns in the relations between words and capture their relations in an embedding space. Schramowski and colleagues show that a direction in this space can be identified that separates ‘right’ and ‘wrong’ actions as judged by human survey participants.

Subfields: Artificial Intelligence, Large Language Models, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Safety, Robustness & Interpretability

Citations in the corpus

An Empirical Study of Metrics to Measure Representational Harms in Pre-Trained Language Models (2023, background): “Therefore, similar to Schramowski et al. (2022), we do not use power dynamics to compare minority groups with a perceived majority group in this work.”
GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models (2023, background): “…partly because out-of-the-box general-purpose LLMs may continue to be unreliable for various tasks due to issues such as factual inaccuracies, inherent biases, privacy concerns, and disinformation risks (Abid et al., 2021; Schramowski et al., 2022; Goldstein et al., 2023; OpenAI, 2023a).”
ChatGPT: Jack of all trades, master of none (2023, background): “Language models are prone to generating responses containing human-like biases as well as presenting moral and ethical stances [23].”
The Tail Wagging the Dog: Dataset Construction Biases of Social Bias Benchmarks (2022, background): “, 2021) and expressed in specific tasks (Nangia et al., 2020; Kirk et al., 2021; Schramowski et al., 2022; Prabhumoye et al., 2021; Srinivasan and Bisk, 2021; Kirk et al., 2021; Parrish et al., 2021; Baldini et al., 2022; Czarnowska et al., 2021; Dev et al., 2021a; Zhao et al., 2021).”
Generative AI (2023, neutral): “Addressing bias and thus fairness in AI receives increasing attention in the academic literature (DeArteaga et al., 2022; Dolata et al., 2022; Ferrara, 2023; Feuerriegel et al., 2020; Schramowski et al., 2022; von Zahn et al., 2022), but remains an open and ongoing research question.”
Practical and ethical challenges of large language models in education: A systematic scoping review (2023, methodology): “, gender and racial biases) when trained on unfiltered internet text data (Schramowski et al., 2022).”
Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models (2022, methodology): “We acknowledge that large language models have been shown to suffer from similar biases to humans [73]; this is to be expected as they have been trained with human-generated data.”
Vygotskian Autotelic Artificial Intelligence: Language and Culture Internalization for Human-Like AI (2022, background): “All this information may already be captured in existing llm s, 51,130 , we may just need to learn to extract it.”
Gender bias and stereotypes in Large Language Models (2023, background): “we find that the models’ behavior tracks people’s beliefs about gender stereotypes concerning occupations more closely than it does the actual ground truth about this distribution as reflected in the BLS statistics [81, 83].”
Generative artificial intelligence (2023, background): “…innovation opportunities for various domains (e.g., networked businesses and digital platforms) but also comes with challenges (e.g., transparency, biases, and misuse) that need to be addressed for successful implementations (Houde et al., 2020; Schramowski et al., 2022; van Slyke et al., 2023).”
The future landscape of large language models in medicine (2023, background): “LLMs have been shown to reproduce existing biases and are susceptible to hallucinating false information and spreading misinformation 32,73 .”