Survey Article: Inter-Coder Agreement for Computational Linguistics

By Ron Artstein, Massimo Poesio. 2008

1,832 citations recorded by the source corpus.

Abstract This article is a survey of methods for measuring agreement among corpus annotators. It exposes the mathematics and underlying assumptions of agreement coefficients, covering Krippendorff's alpha as well as Scott's pi and Cohen's kappa; discusses the use of coefficients in several annotation tasks; and argues that weighted, alpha-like coefficients, traditionally less used than kappa-like measures in computational linguistics, may be more appropriate for many corpus annotation tasks—but that their use makes the interpretation of the value of the coefficient even harder.

Subfields: Artificial Intelligence, Management Science and Operations Research, Statistics and Probability, Evaluation & Benchmarks, Computational Linguistics & Semantics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Noah A. Smith
  2. Jianfeng Gao
  3. Mohit Bansal
  4. Yejin Choi
  5. Luke S. Zettlemoyer
  6. Percy Liang
  7. Graham Neubig
  8. Kyunghyun Cho

In Management Science and Operations Research

  1. Alistair Moffat
  2. David Schlangen
  3. Iryna Gurevych
  4. Verena Rieser
  5. Christian Stab
  6. Yvette Graham
  7. William B. Dolan
  8. David Jurgens

In Statistics and Probability

  1. David Schlangen
  2. Iryna Gurevych
  3. S. Gadarian
  4. B. Albertson
  5. Christian Stab
  6. William B. Dolan
  7. David Jurgens
  8. Dani Yogatama

In Evaluation & Benchmarks

  1. Elizabeth Clark
  2. Yvette Graham
  3. Ondrej Bojar
  4. Verena Rieser
  5. Jianfeng Gao
  6. Noah A. Smith
  7. Lucia Specia
  8. Anya Belz

In Computational Linguistics & Semantics

  1. David Schlangen
  2. Iryna Gurevych
  3. Christian Stab
  4. Noah A. Smith
  5. William B. Dolan
  6. David Jurgens
  7. Steven Bedrick
  8. Ivan Habernal

Citations in the corpus