Bias decreases in proportion to the number of annotators

By Ron Artstein, Massimo Poesio. 2005

50 citations recorded by the source corpus.

Subfields: Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

1. Elizabeth Clark
2. Sebastian Gehrmann
3. Thibault Sellam
4. Craig Thomson
5. Emiel van Miltenburg
6. Ondrej Dusek
7. Miruna Clinciu
8. S. Schoch

Citations in the corpus

• Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “the overall bias (Artstein and Poesio, 2009), this finding assumes that the population of annotators is somehow representative of the whole world. Prabhakaran et al. (2021) find that aggregating annota-”
• Underreporting of errors in NLG output, and what to do about it (2021, background): “Generally speaking, having more annotators reduces the prevalence of the individual bias (Artstein and Poesio, 2005).”
• Survey Article: Inter-Coder Agreement for Computational Linguistics (2008, unknown): “But we also believe that ultimately this issue of annotator bias is of little consequence because the differences get smaller and smaller as the number of annotators grows (Artstein and Poesio 2005).”