Building Better Open-Source Tools to Support Fairness in Automated Scoring

By Nitin Madnani, Anastassia Loukina, Alina von Davier, Jill Burstein, Aoife Cahill. 2017

40 citations recorded by the source corpus.

Automated scoring of written and spoken responses is an NLP application that can significantly impact lives especially when deployed as part of high-stakes tests such as the GRE® and the TOEFL®. Ethical considerations require that automated scoring algorithms treat all test-takers fairly. The educational measurement community has done significant research on fairness in assessments and automated scoring systems must incorporate their recommendations. The best way to do that is by making available automated, non-proprietary tools to NLP researchers that directly incorporate these recommendations and generate the analyses needed to help identify and resolve biases in their scoring systems. In this paper, we attempt to provide such a solution.

Subfields: Management Science and Operations Research, Safety Research, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Management Science and Operations Research

1. Ezekiel Dixon-Román
2. Alan W. Black
3. Hal Daumé Iii
4. Nitin Madnani
5. Hanna M. Wallach
6. Anastassia Loukina
7. Solon Barocas
8. Klaus Zechner

In Safety Research

1. Hanna M. Wallach
2. Su Lin Blodgett
3. Alan W. Black
4. Hal Daumé Iii
5. Solon Barocas
6. Adrian Weller
7. Alexandra Chouldechova
8. Berk Ustun

In Natural Language Processing

1. Hal Daumé Iii
2. Shrimai Prabhumoye
3. Su Lin Blodgett
4. Alan W. Black
5. Nitin Madnani
6. Hanna M. Wallach
7. Solon Barocas
8. Elijah Mayfield

In Safety, Robustness & Interpretability

1. Su Lin Blodgett
2. Hal Daumé Iii
3. Hanna M. Wallach
4. Shrimai Prabhumoye
5. Alan W. Black
6. Nitin Madnani
7. Anastassia Loukina
8. Solon Barocas

Citations in the corpus

• Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, background): “A.2 Full categorization: Motivations Allocational harms Hovy and Spruit (2016); Caliskan et al. (2017); Madnani et al. (2017); Dixon et al. (2018); Kiritchenko and Mohammad (2018); Shen et al. (2018); Zhao et al. (2018b); Bhaskaran and Bhallamudi (2019); Bordia and Bowman (2019); Brunet et al.…”
• Equity Beyond Bias in Language Technologies for Education (2019, methodology): “Madnani et al. (2017), for instance, presents a capable tool for evaluating fair outcomes in automated essay scoring.”
• The many dimensions of algorithmic fairness in educational applications (2019, methodology): “The model was fitted using Ridge regression available via RSMTool, an open-source tool for training automated scoring models (Madnani et al., 2017).”
• An automated essay scoring systems: a systematic literature review (2021, unknown): “From Loukina et al. (2019), and Madnani et al. (2017b).”