Understanding interobserver agreement: the kappa statistic.

By A. Viera, J. Garrett. 2005

7,776 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Text Understanding from Scratch (2015, background): “Many researchers have shown that with some random text, the inter-rater consensus on polarity is only about 60% 80%(Gamon & Aue, 2005)(Kim & Hovy, 2004)(Strapparava & Mihalcea, 2008)(Viera et al., 2005)(Wiebe et al.”
The Values Encoded in Machine Learning Research (2021, neutral): “61, which indicates substantial agreement [39].”
Patient engagement in research: a systematic review (2014, background): “80) [13] was achieved, after reviewing 200 potentially includible references.”
The Effects of Providing and Receiving Peer Feedback on Writing Performance and Learning of Secondary School Students (2020, background): “The interrater reliability was substantial (Anthony & Joanne, 2005) for both high-level scores (k = .73) and low-level scores (k = .70).”
Artie Bias Corpus: An Open Dataset for Detecting Demographic Bias in Speech Applications (2020, unknown): “As per Viera et al. (2005), we reached “fair” to “moderate” agreement.”
A comprehensive study on deep learning bug characteristics (2019, methodology): “We measured the inter rater aggrement among the labellers using Cohen’s Kappa coefficient [22] when 5%, 10%, 20% , 30%, 40%, 50%, 60%, 70%, 80%, 90% and 100% of the posts were labeled.”
Effect of eWOM stimuli and eWOM response on perceived service quality and online recommendation (2020, result): “…ensure the inter-coder reliability and consistency of coding the hotel reviews in terms of their eWOM valence, volume and image-based responses, a kappa statistics result of 0.80 reflected substantial agreements between the coders for listing the data for the select hotels (Viera & Garrett, 2005).”
The Creation and Analysis of a Website Privacy Policy Corpus (2016, background): “Since Kappa is applied here to an artificial task (annotators were not asked to label entire segments) the common conventions for its interpretation (Carletta, 1996; Viera and Garrett, 2005) are not directly applicable.”
Change in terrestrial human footprint drives continued loss of intact ecosystems (2020, background): “systems.(4,5) As a consequence, calls for the global identification, monitoring, and retention of the remaining lands that are relatively free of direct anthropogenic disturbance are increasing.”
Mean Birds: Detecting Aggression and Bullying on Twitter (2017, background): “22, which indicates a fair level of agreement between our workers [55].”
Pharmacovigilance from social media: mining adverse drug reaction mentions using sequence labeling with word embedding cluster features (2015, background): “40 Finally, the gold standard was generated by including only the reviews with complete inter-annotator agreement.”
Improving fluid intelligence with training on working memory: a meta-analysis (2014, unknown): “This falls in the “ almost perfect agreement ” range (Viera & Garrett, 2005).”
Flipped classroom in English language teaching: a systematic review (2020, unknown): “According to Viera and Garrett (2005), Figure 1.”