Persistent Anti-Muslim Bias in Large Language Models

By Abubakar Abid, Maheen Farooqi, James Y. Zou. 2021

721 citations recorded by the source corpus.

It has been observed that large-scale language models capture undesirable societal biases, e.g. relating to race and gender; yet religious bias has been relatively unexplored. We demonstrate that GPT-3, a state-of-the-art contextual language model, captures persistent Muslim-violence bias. We probe GPT-3 in various ways, including prompt completion, analogical reasoning, and story generation, to understand this anti-Muslim bias, demonstrating that it appears consistently and creatively in different uses of the model and that it is severe even compared to biases about other religious groups. For instance, Muslim is analogized to terrorist in 23% of test cases, while Jewish is mapped to its most common stereotype, money, in 5% of test cases. We quantify the positive distraction needed to overcome this bias with adversarial text prompts, and find that use of the most positive 6 adjectives reduces violent completions for Muslims from 66% to 20%, but which is still higher than for other religious groups.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, critical): “In particular we see in Table 37 that Islam is associated with highly charged terms and stereotypes such as terrorist, violent and radical, similar to other evaluations of decoder-only models revealing anti-Muslim bias (Abid et al., 2021).”
Challenges in Detoxifying Language Models (2021, background): “…capture less obvious, but no less important harms—such as subtle or distributional biases (Sap et al., 2019b; Sheng et al., 2019; Huang et al., 2020; Abid et al., 2021). by steering a model’s generation towards text less likely to be classified as toxic (Dathathri et al., 2020; Krause et al.,…”
Evaluating Large Language Models Trained on Code (2021, background): “Mirroring what has been found in the case of other language models trained on Internet data (Bender et al., 2021; Blod-gett et al., 2020; Abid et al., 2021; Brown et al., 2020”
Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets (2021, background): “…due in no small part to the deeply contextual and open nature of language use and tasks (Austin, 1975; Clark, 1996; Howcroft et al., 2020; Abid et al., 2021; Olteanu et al., 2020), and a lack of consensus about how to conceptualize or operationalize such harms (Blod-gett et al., 2020;…”
Pretraining Language Models with Human Preferences (2023, background): “Toxicity LMs can generate highly harmful language, including insults, profanities and threats (Sap et al., 2019; Gehman et al., 2020; Abid et al., 2021).”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “(2) Toxicity is generated language that is rude, disrespectful, offensive, threatening, violent, or otherwise harmful (Gehman et al., 2020; Rae et al., 2021; Abid et al., 2021).”
Re-contextualizing Fairness in NLP: The Case of India (2022, background): “Religion: Religious biases have been studied in NLP (Dev et al., 2020; Nadeem et al., 2020; Abid et al., 2021), however the social disparities and stereotypes about various religious groups differ significantly in India from the West, (Malik et al., 2021).”
Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets (2021, background): “[4] Earlier analyses of harmful outputs in GPT-3 show negative race, gender[8], and religious[3] associations in generated text.”
GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models (2023, background): “…partly because out-of-the-box general-purpose LLMs may continue to be unreliable for various tasks due to issues such as factual inaccuracies, inherent biases, privacy concerns, and disinformation risks (Abid et al., 2021; Schramowski et al., 2022; Goldstein et al., 2023; OpenAI, 2023a).”
The ROOTS Search Tool: Data Transparency for LLMs (2023, background): “For example, GPT-3 has been shown to over-associate Muslims with violence (Abid et al., 2021).”
A Hazard Analysis Framework for Code Synthesis Large Language Models (2022, background): “• Potential inline generation features being utilized in an open-ended manner permitting general, non-code usage of language model capabilities, mirroring what has been found in the case of other language models trained on Internet data [2, 5, 7, 10].”
Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP (2021, background): “…et al., 2016; Caliskan et al., 2017; Zhao et al., 2017; Rudinger et al., 2018; Gonen and Goldberg, 2019; Bordia and Bowman, 2019; Sheng et al., 2019; Basta et al., 2019; Nangia et al., 2020, i.a. ) and are prone to generating toxic texts (Brown et al., 2020; Gehman et al., 2020; Abid et al., 2021).”
What’s in the Box? An Analysis of Undesirable Content in the Common Crawl Corpus (2021, background): “…larger corpora generally result in better models (Kaplan et al., 2020; Sun et al., 2017), data quality and corpora content also plays a major role in the caliber and appropriateness of these models for the various downstream applications (Florez, 2019; Abid et al., 2021; Bhardwaj et al., 2021).”
You reap what you sow: On the Challenges of Bias Evaluation Under Multilingual Settings (2022, background): “Although word embeddings may help identify biases in the context of LLMs, it is often difﬁcult to access the learned contextual language representations of the model (Abid et al., 2021; Dev et al., 2020).”
What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers (2021, background): “Many studies have already reported that these biases from training data have signiﬁcant inﬂuences on large-scale language models as well (Abid et al., 2021; Garrido-Muñoz et al., 2021; Shwartz and Choi, 2020).”
Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models (2022, background): “…[108] Gender Gender & Occ [21, 85] Gender Deconfounding [43] Gender, Profession, Country TruthfulQA [74] None DTC [71] Gender, Religion Muslim Bias [5] Religion BAD[107] None* BOLD [37] Profession, Gender, Race, Religion, Political Ideology Stereoset [78] Gender, Profession, Race, Religion…”
Reason first, then respond: Modular Generation for Knowledge-infused Dialogue (2021, background): “On a more fundamental note, language models suffer from biases in the training data (Lu et al., 2020; Abid et al., 2021), and can generate unsafe or even toxic language when prompted with the wrong context (Roller et al., 2021).”