Toward Deconfounding the Effect of Entity Demographics for Question Answering Accuracy
By Maharshi Gor, Kellie Webster, Jordan L. Boyd-Graber. 2021
11 citations recorded by the source corpus.
The goal of question answering (QA) is to answer _any_ question. However, major QA datasets have skewed distributions over gender, profession, and nationality. Despite that skew, an analysis of model accuracy reveals little evidence that accuracy is lower for people based on gender or nationality; instead, there is more variation on professions (question topic) and question ambiguity. But QA’s lack of representation could itself hide evidence of bias, necessitating QA datasets that better represent global diversity.
Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Percy Liang
- Caiming Xiong
- Richard Socher
- Yoshua Bengio
- Sergey Levine
- Luke S. Zettlemoyer
In Information Systems
In Natural Language Processing
- Mohit Bansal
- Jianfeng Gao
- Po-Sen Huang
- Geoffrey Irving
- Iason Gabriel
- Abigail See
- John F. J. Mellor
- Susannah Young
In Safety, Robustness & Interpretability
- Iason Gabriel
- Po-Sen Huang
- Boxi Wu
- William S. Isaac
- John F. J. Mellor
- Nat McAleese
- Johannes Welbl
- Geoffrey Irving
Citations in the corpus
- Improving alignment of dialogue agents via targeted human judgements (2022, methodology): “Following Gor et al. (2021), we use a 𝑝-value threshold of 0.05 and divide it by three, as we have three tests for each dataset.”
- Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models (2022, background): “…[15] Speakers of AAE SAE/AAVE Pairs [44] Speakers of AAVE Winogender [87] Gender Winobias [108] Gender Gender & Occ [21, 85] Gender Deconfounding [43] Gender, Profession, Country TruthfulQA [74] None DTC [71] Gender, Religion Muslim Bias [5] Religion BAD[107] None* BOLD [37] Profession, Gender,…”