"Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letters

By Yixin Wan, George Pu, Jiao Sun, Aparna Garimella, Kai-Wei Chang, Nanyun Peng. Conference on Empirical Methods in Natural Language Processing, 2023

332 citations recorded by the source corpus.

Large Language Models (LLMs) have recently emerged as an effective tool to assist individuals in writing various types of content, including professional documents such as recommendation letters. Though bringing convenience, this application also introduces unprecedented fairness concerns. Model-generated reference letters might be directly used by users in professional scenarios. If underlying biases exist in these model-constructed letters, using them without scrutinization could lead to direct societal harms, such as sabotaging application success rates for female applicants. In light of this pressing issue, it is imminent and necessary to comprehensively study fairness issues and associated harms in this real-world use case. In this paper, we critically examine gender biases in LLM-generated reference letters. Drawing inspiration from social science findings, we design evaluation methods to manifest biases through 2 dimensions: (1) biases in language style and (2) biases in lexical content. We further investigate the extent of bias propagation by analyzing the hallucination bias of models, a term that we define to be bias exacerbation in model-hallucinated contents. Through benchmarking evaluation on 2 popular LLMs- ChatGPT and Alpaca, we reveal significant gender biases in LLM-generated recommendation letters. Our findings not only warn against using LLMs for this application without scrutinization, but also illuminate the importance of thoroughly studying hidden biases and harms in LLM-generated professional documents.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/ce157cea880c9ab64de64f11a531202f5348fa05

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Jiayi Ye
Yanbo Wang
Yue Huang
Qihui Zhang
Nuno Moniz
Tian Gao
Werner Geyer
Chao Huang

Citations in the corpus

Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge (2024, background): “The imbalance in pre-training data can lead to imbalances during model training (Liu et al., 2024), resulting in biases against certain demographic groups, such as different genders (Wan et al., 2023), ages (Macnicol, 2006), and various languages (Jiao et al., 2023; Bang et al., 2023).”
A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly (2023, background): “In this section, we examine the range of existing defense methods against various attacks and vulnerabilities associated with LLMs 1 .”
When LLMs Play the Telephone Game: Cultural Attractors as Conceptual Tools to Evaluate LLMs in Multi-turn Settings (2024, unknown): “In these studies, LLMs showed several biases with respect to gender (Acerbi and Stubbersfield, 2023; Salewski et al., 2024; Wan et al., 2023; Haller et al., 2023; Dong et al., 2024), race (Raj et al., 2024), values (Atari et al., 2023; Sivaprasad et al., 2024), politics (Motoki et al., 2023; Agiza…”