ORES

By Aaron L Halfaker, R. Stuart Geiger. 2019

20 citations recorded by the source corpus.

Algorithmic systems---from rule-based bots to machine learning classifiers---have a long history of supporting the essential work of content moderation and other curation work in peer production projects. From counter-vandalism to task routing, basic machine prediction has allowed open knowledge projects like Wikipedia to scale to the largest encyclopedia in the world, while maintaining quality and consistency. However, conversations about how quality control should work and what role algorithms should play have generally been led by the expert engineers who have the skills and resources to develop and modify these complex algorithmic systems. In this paper, we describe ORES: an algorithmic scoring service that supports real-time scoring of wiki edits using multiple independent classifiers trained on different datasets. ORES decouples several activities that have typically all been performed by engineers: choosing or curating training data, building models to serve predictions, auditing predictions, and developing interfaces or automated agents that act on those predictions. This meta-algorithmic system was designed to open up socio-technical conversations about algorithms in Wikipedia to a broader set of participants. In this paper, we discuss the theoretical mechanisms of social change ORES enables and detail case studies in participatory machine learning around ORES from the 5 years since its deployment.

Subfields: Communication, Computer Science Applications, Information Systems, Natural Language Processing, AI Ethics, Fairness & Society

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Communication

1. William S. Isaac

In Computer Science Applications

1. Jonathan Uesato

In Information Systems

1. William S. Isaac
2. Iason Gabriel
3. Po-Sen Huang
4. Doug Fritz
5. Fan Yang
6. Jonathan Uesato

In Natural Language Processing

1. Po-Sen Huang
2. Geoffrey Irving
3. Jianfeng Gao
4. Mohit Bansal
5. Iason Gabriel
6. Abigail See
7. John F. J. Mellor
8. Susannah Young

In AI Ethics, Fairness & Society

1. William S. Isaac
2. Iason Gabriel

Citations in the corpus

• Improving alignment of dialogue agents via targeted human judgements (2022, background): “In future, more participatory inputs (Berditchevskaia et al., 2021; Halfaker and Geiger, 2020; Lee et al., 2019) from other stakeholders will be critical for developing language agents that are both legitimate and aligned to the needs of its users.”