On Measuring Gender Bias in Translation of Gender-neutral Pronouns

By Won Ik Cho, Jiwon Kim, Seokhwan Kim, Nam Soo Kim. 2019

98 citations recorded by the source corpus.

Ethics regarding social bias has recently thrown striking issues in natural language processing. Especially for gender-related topics, the need for a system that reduces the model bias has grown in areas such as image captioning, content recommendation, and automated employment. However, detection and evaluation of gender bias in the machine translation systems are not yet thoroughly investigated, for the task being cross-lingual and challenging to define. In this paper, we propose a scheme for making up a test set that evaluates the gender bias in a machine translation system, with Korean, a language with gender-neutral pronouns. Three word/phrase sets are primarily constructed, each incorporating positive/negative expressions or occupations; all the terms are gender-independent or at least not biased to one side severely. Then, additional sentence lists are constructed concerning formality of the pronouns and politeness of the sentences. With the generated sentence set of size 4,236 in total, we evaluate gender bias in conventional machine translation systems utilizing the proposed measure, which is termed here as translation gender bias index (TGBI). The corpus and the code for evaluation is available on-line.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, background): “…Kiritchenko and Mohammad (2018); Lu et al. (2018); Rudinger et al. (2018); Shen et al. (2018); Bordia and Bowman (2019); Cao and Daumé (2019); Cho et al. (2019); Davidson et al. (2019); Dev et al. (2019); Dinan et al. (2019); Fisher (2019); Florez (2019); Font and Costa-jussà (2019); Garg et…”
Societal Biases in Language Generation: Progress and Challenges (2021, background): “While Google Translate 3 has been the most popular commercial system to analyze for gender biases (Prates et al., 2019; Moryossef et al., 2019; Stanovsky et al., 2019; Cho et al., 2019; Farkas and N´emeth, 2020), Stanovsky et al. (2019) 2021) also examine Yandex.”
Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models (2021, background): “, 2021), neural machine translation (Cho et al., 2019; Stanovsky et al., 2019; Prates et al., 2020; Renduchintala et al., 2021; Savoldi et al., 2021), or language modeling (Nangia et al.”
Towards Understanding and Mitigating Social Biases in Language Models (2021, background): “, 2017), and questionable correlations between system behavior and features associated with particular social groups (Cho et al., 2019).”
UNQOVERing Stereotypical Biases via Underspecified Questions (2020, background): “, 2019), machine translation (Stanovsky et al., 2019; Cho et al., 2019), textual entailment (Dev et al.”
Towards Cross-Lingual Generalization of Translation Gender Bias (2021, methodology): “Terms that do not meet the ethical standard applied to sentiment words were removed referring to Cho et al. [6], that is, those related to appearance, economic status, sexuality, level of education, disability, etc.”
Evaluating Gender Bias in Hindi-English Machine Translation (2021, methodology): “Considering all of the requirements laid out by Cho et al. (2019), we created a list of unique occupations and positive and negative sentiment in our source language, Hindi.”
Improving Gender Translation Accuracy with Filtered Self-Training (2021, unknown): “1 For example, on input sentences that are underspeciﬁed in terms of gender, models often default to masculine or gender-stereotypical outputs (Cho et al., 2019; Prates et al., 2018), which can have the effect of excluding female and non-binary people (e.g. the sentence I am a doctor spoken by a…”
Undesirable biases in NLP: Averting a crisis of measurement (2022, background): “In Korean and Hungarian he/she are gender-neutral and in German, she can also signify they, them or you (Cho et al., 2019; Prates et al., 2020; Chen et al., 2021).”
Gender Bias in Machine Translation (2021, background): “Bureau of Labor Statistics b R: under-rep, stereotyping (Cho et al., 2019) Synthetic equity evaluation corpus (EEC) b R: under-rep, stereotyping”
Automatically Identifying Gender Issues in Machine Translation using Perturbations (2020, methodology): “We extend this line of work, which identifies issues using synthetic examples manually curated for a target language (Stanovsky et al., 2019; Cho et al., 2019), by analyzing real world text across a range of languages to understand challenges for deployed systems.”
How to Measure Gender Bias in Machine Translation: Optimal Translators, Multiple Reference Points (2020, background): “Comparing Google Translate’s results to all three types of optimal translators, we confirmed the findings of previous studies that discovered gender bias in the algorithm (Cho et al., 2019; Prates et al., 2020).”
A Survey on Gender Bias in Natural Language Processing (2021, methodology): “However, we note that these methods do not inherently remove gender bias from the system [Cho et al. 2019].”