What Determines Inter-Coder Agreement in Manual Annotations? A Meta-Analytic Investigation

By Petra Saskia Bayerl, Karsten Ingmar Paul. 2011

129 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jean–Marc Dewaele
2. Alistair Willis
3. Samuel Greiff
4. Paul Piwek
5. Jacopo Amidei
6. E. Botes
7. Elizabeth Clark
8. Hung-yi Lee

In Natural Language Processing

1. Alistair Willis
2. Paul Piwek
3. Jacopo Amidei

In Evaluation & Benchmarks

1. Alistair Willis
2. Paul Piwek
3. Jacopo Amidei
4. Elizabeth Clark
5. Hung-yi Lee
6. Cheng-Han Chiang
7. Jianfeng Gao
8. Asli Celikyilmaz

Citations in the corpus

• The Foreign Language Classroom Anxiety Scale and Academic Achievement: An Overview of the Prevailing Literature and a Meta-analysis (2020, background): “Twenty of the 64 studies were coded by two coders, with the proportion of the number of observations agreed upon at an acceptable level of AR = 90% (Bayerl & Paul, 2011).”
• Rethinking the Agreement in Human Evaluation Tasks (2018, methodology): “Here we are following the notation used in Carletta (1996). It is worth notice that the K formulation presented catch the most used agreement metrics from the NLP community, such as the π coefficient of Scott (1955) its Fleiss’ generalization (Fleiss, 1971), and the k coefficient of Cohen (1960). Th”