Gender Coreference and Bias Evaluation at WMT 2020

By Tom Kocmi, Tomasz Limisiewicz, Gabriel Stanovsky. 2020

33 citations recorded by the source corpus.

Gender bias in machine translation can manifest when choosing gender inflections based on spurious gender correlations. For example, always translating doctors as men and nurses as women. This can be particularly harmful as models become more popular and deployed within commercial systems. Our work presents the largest evidence for the phenomenon in more than 19 systems submitted to the WMT over four diverse target languages: Czech, German, Polish, and Russian. To achieve this, we use WinoMT, a recent automatic test suite which examines gender coreference and bias when translating from English to languages with grammatical gender. We extend WinoMT to handle two new languages tested in WMT: Polish and Czech. We find that all systems consistently use spurious correlations in the data rather than meaningful contextual information.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Societal Biases in Language Generation: Progress and Challenges (2021, background): “…to study gender biases when translating from neutral references of a person (e.g., nurse in English, gender-neutral pronouns) to gendered instances (e.g., enfermera or enfermero in Spanish, gendered pronouns) (Cho et al., 2019; Stanovsky et al., 2019; Gonen and Webster, 2020; Kocmi et al., 2020).”
What do Bias Measures Measure? (2021, background): “There are also extensions of WinoMT for different languages (Kocmi et al., 2020) and datasets collected through mining (Gonen and Web-ster, 2020).”
Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translation (2021, background): “Several works created synthetic datasets to evaluate gender bias (Kiritchenko and Mohammad, 2018; González et al., 2020; Renduchintala and Williams, 2021), e.g., in the context of coreference (Rudinger et al., 2017; Zhao et al., 2018) and machine translation (Stanovsky et al., 2019; Prates et al.,…”
Findings of the 2020 Conference on Machine Translation (WMT20) (2020, background): “This year all MT systems fall into the gender bias trap (Kocmi et al., 2020) and they tend to make more severe errors than humans (Zouhar et al.”