Estimating Interrater Agreement with the Average Deviation Index: A User’s Guide

By Michael J. Burke, W. Dunlap. 2002

488 citations recorded by the source corpus.

Subfields: Management Science and Operations Research, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/89319bcb1bed2a3e9f8adde082ed27740eca9bff

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Management Science and Operations Research

In Statistics and Probability

Citations in the corpus

The Social Consequences of Voice: An Examination of Voice Type and Gender on Status and Subsequent Leader Emergence (2017, methodology): “To assess rater agreement, we followed recommendations for ranking-type data by Burke and Dunlap (2002) to calculate median average deviation (ADmed), and compare our results against a uniform distribution, which is the distribution that would result if every ranking were given by chance.”
Reliability and agreement of student ratings of the classroom environment: A reanalysis of TIMSS data (2007, methodology): “(1999) and Burke and Dunlap (2002). The average deviation index ADM is defined by the deviations of the raters from their group mean:”
The role of national culture and organizational climate in safety training effectiveness (2008, methodology): “This result is well below the cutoff value for acceptable interrater agreement using a 5-point Likert-type scale (i.e., of .8, where ADM values of zero indicate perfect agreement; see Burke & Dunlap, 2002).”
A Sequential Analysis of Procedural Meeting Communication: How Teams Facilitate Their Meetings (2013, methodology): “Compared to other measures of dispersion, this measure is more easily interpretable from original scores (Burke & Dunlap, 2002; Dawson, Gonzalez-Roma, Davis, & West, 2008).”
What do the young (old) people think of me? Content and accuracy of age-based metastereotypes (2013, methodology): “Accordingly, we calculated Burke and Dunlap's ( 2002 ) absolute average deviation index in order to ensure agreement among raters.”
EMOTIONAL INTELLIGENCE AND LEADERSHIP EMERGENCE IN SMALL GROUPS (2010, background): “67 recommended by Burke and Dunlap (2002) and, hence, it reveals high agreement among raters.”
Rocking the Boat but Keeping It Steady: The Role of Emotion Regulation in Employee Voice (2013, methodology): “…intraclass correlation coefficients, and interrater agreement using average deviation (AD), which “estimates agreement in the metric of the original scale of the item” (LeBreton & Senter, 2008: 820), whereby AD values below 1.2 indicate high agreement for sevenpoint scales (Burke & Dunlap, 2002).”
A SITUATIONAL JUDGMENT TEST OF PERSONAL INITIATIVE AND ITS RELATIONSHIP TO PERFORMANCE (2009, background): “Burke and Dunlap (2002) have provided a criterion for acceptable levels of agreement.”
Organizational readiness for implementing change: a psychometric assessment of a new measure (2014, background): “33, which is much lower t Therefore, the conclusion is that based on the mean rWG(J) we reject the null hypoth both change commitment and change efficacy [15,27].”
Interrater Agreement Reconsidered: An Alternative to the rwg Indices (2005, background): “(Burke and colleagues [Burke & Dunlop, 2002; Burke, Finklestein, & Dusig, 1999] recently proposed an interrater agreement index based on average deviations from the mean/median.)”