Bias and Fairness in Large Language Models: A Survey

By Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, Nesreen K. Ahmed. Computational Linguistics, 2023

1,188 citations recorded by the source corpus.

Abstract Rapid advancements of large language models (LLMs) have enabled the processing, understanding, and generation of human-like text, with increasing integration into systems that touch our social sphere. Despite this success, these models can learn, perpetuate, and amplify harmful social biases. In this article, we present a comprehensive survey of bias evaluation and mitigation techniques for LLMs. We first consolidate, formalize, and expand notions of social bias and fairness in natural language processing, defining distinct facets of harm and introducing several desiderata to operationalize fairness for LLMs. We then unify the literature by proposing three intuitive taxonomies, two for bias evaluation, namely, metrics and datasets, and one for mitigation. Our first taxonomy of metrics for bias evaluation disambiguates the relationship between metrics and evaluation datasets, and organizes metrics by the different levels at which they operate in a model: embeddings, probabilities, and generated text. Our second taxonomy of datasets for bias evaluation categorizes datasets by their structure as counterfactual inputs or prompts, and identifies the targeted harms and social groups; we also release a consolidation of publicly available datasets for improved access. Our third taxonomy of techniques for bias mitigation classifies methods by their intervention during pre-processing, in-training, intra-processing, and post-processing, with granular subcategories that elucidate research trends. Finally, we identify open problems and challenges for future work. Synthesizing a wide range of recent research, we aim to provide a clear guide of the existing literature that empowers researchers and practitioners to better understand and prevent the propagation of bias in LLMs.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/bcfa73aedf1b2d1ee4f168e21298a37ac55a37f7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model (2024, background): “It is also important to keep in mind that adversarial use for intentional harm, as mitigated here, makes up only one specific aspect of LLM Safety (Bender et al., 2021; Gallegos et al., 2023; Huang et al., 2023b; Li et al., 2023d), and that safety measures have to get extended beyond that.”
A Survey on the Memory Mechanism of Large Language Model-based Agents (2024, neutral): “Gödel Agent [206] utilizes dynamic code modification techniques to alter its own logic and behavior based on the feedback signals from the environment and stores the updated agent logic in the agent memory for recursive improvement.”
Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models (2024, result): “Studies in this area, as evidenced by the work of Gallegos et al. [126], Zhang et al. [127], Liang et al. [128], and Friedrich et al. [129], are dedicated to the meticulous identification and rectification of these inherent biases.”
Towards conversational diagnostic artificial intelligence (2025, positive): “Collectively, our findings suggest many avenues for further research that might leverage human–AI complementarity 30 , combining clinicians’ skills in the analysis of verbal and non-verbal cues with the potential strengths of LLMs to suggest more enriched conversational responses, including empathic”
S-DAT: A Multilingual, GenAI-Driven Framework for Automated Divergent Thinking Assessment (2025, unknown): “In doing so, S-DAT contributes to a broader shift toward equitable, globally relevant AI research—aligned with current debates on language bias and representational fairness in AI systems (Hou and Huang, 2025; Gallegos et al., 2024).”