Answers to 20 Questions About Interrater Reliability and Interrater Agreement

By James M. LeBreton, Jenell L. Senter. 2008

3,557 citations recorded by the source corpus.

Subfields: Management Science and Operations Research, Organizational Behavior and Human Resource Management

View Paper: https://www.semanticscholar.org/paper/a27d2559be1b48ca8ce31eeaa514943303495f75

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Management Science and Operations Research

1. Steve W. J. Kozlowski
2. J. Edwards
3. Robert J. Vandenberg
4. Adam M. Grant
5. John E. Mathieu
6. Michael S. Cole
7. Lawrence J. Williams
8. Wilmar B. Schaufeli

In Organizational Behavior and Human Resource Management

1. Arnold B. Bakker
2. Evangelia Demerouti
3. Sharon K. Parker
4. Wilmar B. Schaufeli
5. Sabine Sonnentag
6. Thomas W. H. Ng
7. Adam M. Grant
8. Christian Vandenberghe

Citations in the corpus

• Leader Humility and Team Performance: Exploring the Mediating Mechanisms of Team PsyCap and Task Allocation Effectiveness (2019, neutral): “%) represent “strong” and “very strong” interrater agreement (LeBreton & Senter, 2008: 836), respectively.”
• Turning Agents into Psychological Principals: Aligning Interests of Non‐Owners Through Psychological Ownership (2013, neutral): “When one target is assessed by multiple raters with multiple items, such as company performance in our case, the rwg(j) index may provide the necessary empirical support to justify the aggregation of individual-level data to a higher level (e.g. company level) (LeBreton and Senter, 2008).”
• Learning From Mistakes: How Mistake Tolerance Positively Affects Organizational Learning and Performance (2017, methodology): “A wellaccepted measure of agreement—rWG(J) (James, Demaree, & Wolf, 1984; LeBreton & Senter, 2008) was used to justify the aggregation of individual level data in this study. rWG assesses consensus and is used to determine if adequate agreement exists among ratings for any given organization.”
• Does Team Culture Matter? Roles of Team Culture and Collective Regulatory Focus in Team Task and Creative Performance (2016, background): “In addition, given that responses with low within-team agreement preclude capturing shared perceptions of collective phenomena (Choi, 2009; LeBreton & Senter, 2008), we excluded teams whose rwg(j) values were lower than .50 from the sample (27 members of 7 teams).”
• Understanding Why Consumers Don't Skip Pre-Roll Video Ads (2017, methodology): “Following completion of this process, interrater consistency was assessed on a per-ad basis by calculating Pearson correlations (Gwet 2014; LeBreton and Senter 2008).”
• Innovative Behaviour, Trust and Perceived Workplace Performance (2018, neutral): “80 indicate acceptable levels of agreement (LeBreton and Senter, 2007).”
• Assembling Fragments Into a Lens: A Review, Critique, and Proposed Research Agenda for the Organizational Work Climate Literature (2009, methodology): “We refer readers to articles such as Bliese (2000) and LeBreton and Senter (2008) for a more in-depth treatment of this issue.”
• Are authentic leaders associated with more virtuous, committed and potent teams? ☆ ☆☆ (2013, methodology): “…higher than the cutoff value of 0.60 suggested by some authors (Michel et al., 2011; Zellmer-Bruhn, 2003), three being higher and one being close to the cutoff value of 0.70 suggested by other authors (Avolio, Gardner, Walumbwa, Luthans, & May, 2004; Biemann et al., 2012; LeBreton & Senter, 2008).”
• Delighting the Customer: Creativity-Oriented High-Performance Work Systems, Frontline Employee Creative Performance, and Customer Satisfaction (2019, result): “As was the case for creative performance reported above, rwg(j) values were all >.70 (HPWS = .96, need satisfaction = .94, and creative process engagement = .95), indicating a strong agreement within the branches (LeBreton & Senter, 2008).”
• How Do Developmental and Accommodative HRM Enhance Employee Engagement and Commitment? The Role of Psychological Contract and SOC Strategies (2013, methodology): “ICC1s are lower than recommended (LeBreton and Senter, 2008), yet this is not problematic when the sample is large, and when the other statistics fulfil the criteria (Klein et al.”
• Dispersion-Composition Models in Multilevel Research (2011, methodology): “Accordingly, LeBreton and Senter (2008) have recommended that ‘‘traditional conventions’’ be adopted when interpreting ICC(1) as an effect size estimate: ‘‘small effect’’ ¼ .01, ‘‘medium effect’’ ¼ .10, and ‘‘large effect’’¼ .25 (p. 838).”