How to Measure Gender Bias in Machine Translation: Optimal Translators, Multiple Reference Points

By A. Farkas, Ren'ata N'emeth. 2020

12 citations recorded by the source corpus.

In this paper, as a case study, we present a systematic study of gender bias in machine translation with Google Translate. We translated sentences containing names of occupations from Hungarian, a language with gender-neutral pronouns, into English. Our aim was to present a fair measure for bias by comparing the translations to an optimal non-biased translator. When assessing bias, we used the following reference points: (1) the distribution of men and women among occupations in both the source and the target language countries, as well as (2) the results of a Hungarian survey that examined if certain jobs are generally perceived as feminine or masculine. We also studied how expanding sentences with adjectives referring to occupations effect the gender of the translated pronouns. As a result, we found bias against both genders, but biased results against women are much more frequent. Translations are closer to our perception of occupations than to objective occupational statistics. Finally, occupations have a greater effect on translation than adjectives.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Societal Biases in Language Generation: Progress and Challenges (2021, background): “While Google Translate3 has been the most popular commercial system to analyze for gender biases (Prates et al., 2019; Moryossef et al., 2019; Stanovsky et al., 2019; Cho et al., 2019; Farkas and Németh, 2020), Stanovsky et al.”