Survey Article: Inter-Coder Agreement for Computational Linguistics
By Ron Artstein, Massimo Poesio. 2008
1,832 citations recorded by the source corpus.
Abstract This article is a survey of methods for measuring agreement among corpus annotators. It exposes the mathematics and underlying assumptions of agreement coefficients, covering Krippendorff's alpha as well as Scott's pi and Cohen's kappa; discusses the use of coefficients in several annotation tasks; and argues that weighted, alpha-like coefficients, traditionally less used than kappa-like measures in computational linguistics, may be more appropriate for many corpus annotation tasks—but that their use makes the interpretation of the value of the coefficient even harder.
Subfields: Artificial Intelligence, Management Science and Operations Research, Statistics and Probability, Evaluation & Benchmarks, Computational Linguistics & Semantics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Noah A. Smith
- Jianfeng Gao
- Mohit Bansal
- Yejin Choi
- Luke S. Zettlemoyer
- Percy Liang
- Graham Neubig
- Kyunghyun Cho
In Management Science and Operations Research
- Alistair Moffat
- David Schlangen
- Iryna Gurevych
- Verena Rieser
- Christian Stab
- Yvette Graham
- William B. Dolan
- David Jurgens
In Statistics and Probability
- David Schlangen
- Iryna Gurevych
- S. Gadarian
- B. Albertson
- Christian Stab
- William B. Dolan
- David Jurgens
- Dani Yogatama
In Evaluation & Benchmarks
- Elizabeth Clark
- Yvette Graham
- Ondrej Bojar
- Verena Rieser
- Jianfeng Gao
- Noah A. Smith
- Lucia Specia
- Anya Belz
In Computational Linguistics & Semantics
- David Schlangen
- Iryna Gurevych
- Christian Stab
- Noah A. Smith
- William B. Dolan
- David Jurgens
- Steven Bedrick
- Ivan Habernal
Citations in the corpus
- Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailment (2018, neutral): “…Bowman et al., 2015; Al Khatib et al., 2016) As the latter metric, many kind of inter-annotator agreement metrics based on multiple human annotation results were employed, because direct assessment of data quality is difficult (Craggs and Wood, 2005; Artstein and Poesio, 2008; Mathet et al., 2012).”
- Twenty Years of Confusion in Human Evaluation: NLG Needs Evaluation Sheets and Standardised Definitions (2020, methodology): “We did this by calculating Krippendorff’s alpha using Jaccard for the distance measure (recommended by Artstein and Poesio 2008).”
- Finding Microaggressions in the Wild: A Case for Locating Elusive Phenomena in Social Media Posts (2019, background): “While lower than what is considered high agreement (Artstein and Poesio, 2008), given the potentially-subjective nature of MA S and criticism for lack of objectivity (Lilienfeld, 2017), we view this result as a strongly positive sign that reliable annotation is possible despite the challenge.”
- Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview (2019, background): “Disagreement between annotators has long been an active research area in NLP, with various approaches to measure and quantify disagreement through inter-annotator agreement (IAA) scores to remove outliers (Artstein and Poesio, 2008).”
- Interpretation of Natural Language Rules in Conversational Machine Reading (2018, background): “This is well within the range of what is considered as substantial agreement (Artstein and Poesio, 2008).”
- Evaluation of Text Generation: A Survey (2020, background): “For an in-depth look at annotator agreement measures in natural language processing, refer to Artstein & Poesio (2008).”
- Sparse Overcomplete Word Vector Representations (2015, background): “An example of an instance is: Table 5: Accuracy of three human annotators on the word intrusion task, along with the average inter-annotator agreement (Artstein and Poesio, 2008) and Fleiss’ κ (Davies and Fleiss, 1982).”
- Neural CRF Model for Sentence Alignment in Text Simplification (2020, neutral): “The inter-annotator agreement is 0.807 measured by Cohen’s kappa (Artstein and Poesio, 2008).”
- A Grain of Salt for the WMT Manual Evaluation (2011, methodology): “Several agreement measures (usually called kappas) were designed based on the Equation 1 (see Artstein and Poesio (2008) and Eugenio and Glass (2004) for an overview and a discussion).”
- The Media Frames Corpus: Annotations of Frames Across Issues (2015, methodology): “ric has been previously recommended for tasks in computational linguistics that involve unitizing (Artstein and Poesio, 2008).”
- Convolutional neural networks applied to high-frequency market microstructure forecasting (2017, background): “provement a classifier gives as a proportion of that of a perfect classifier [20].”
- Error Metrics and Performance Fitness Indicators for Artificial Intelligence and Machine Learning in Engineering and Sciences (2020, methodology): “ssign labels randomly and this is estimated using a per-annotator empirical prior over the class labels. • Measures interannotator agreement. • Expresses the level of agreement between two annotators [109]. • Ranges between -1 and 1. The maximum value means complete agreement. 37 Hamming loss (HL) F”
- Structural Topic Models for Open‐Ended Survey Responses (2014, methodology): “…unleashed on the data and numerical estimates for each document pendix includes estimation details, a comparison to alternative models, a range of simulation studies, and additional tools for applied users. compared across coders (Artstein and Poesio 2008; Lom-bard , Snyder-Duch, and Bracken 2006).”
- Argumentation Mining in User-Generated Web Discourse (2016, methodology): “tem achieves 0.30 score using Krippendorf’s U, which is in the middle between the baseline and the human performance (0.48) but is considered as poor from the inter-annotator agreement point of view (Artstein and Poesio 2008). The boundary similarity metrics is not directly suitable for evaluating a”
- Toxic Language Detection in Social Media for Brazilian Portuguese: New Dataset and Multilingual Analysis (2020, methodology): “Inter- 7https://www.ibge.gov.br/en/home-eng. html annotator agreement is calculated in terms of Krippendorf ’s α (Table 2), since α is robust to multiple annotators, different degrees of disagreement and, missing values (Artstein and Poesio, 2008).”
- Deep learning meets ontologies: experiments to anchor the cardiovascular disease ontology in the biomedical literature (2018, methodology): “…taking into account the categories that appear in the first and second column The fourth column indicates the terms added by the CVDO, when the symbol (R) appears it means that the protein class expressions within the CVDO are used to add terms to the target terms of judgments by all coders ” [96].”
- It’s going to be okay: Measuring Access to Support in Online Communities (2018, positive): “766, indicating substantial agreement on the data (Artstein and Poesio, 2008).”