Death to Kappa: birth of quantity disagreement and allocation disagreement for accuracy assessment
By Robert Gilmore Pontius Jr., Marco Millones. 2011
1,922 citations recorded by the source corpus.
The family of Kappa indices of agreement claim to compare a map's observed classification accuracy relative to the expected accuracy of baseline maps that can have two types of randomness: (1) random distribution of the quantity of each category and (2) random spatial allocation of the categories. Use of the Kappa indices has become part of the culture in remote sensing and other fields. This article examines five different Kappa indices, some of which were derived by the first author in 2000. We expose the indices' properties mathematically and illustrate their limitations graphically, with emphasis on Kappa's use of randomness as a baseline, and the often-ignored conversion from an observed sample matrix to the estimated population matrix. This article concludes that these Kappa indices are useless, misleading and/or flawed for the practical applications in remote sensing that we have seen. After more than a decade of working with these indices, we recommend that the profession abandon the use of Kappa indices for purposes of accuracy assessment and map comparison, and instead summarize the cross-tabulation matrix with two much simpler summary parameters: quantity disagreement and allocation disagreement. This article shows how to compute these two parameters using examples taken from peer-reviewed literature.
Subfields: Artificial Intelligence, Geography, Planning and Development, Signal Processing
View Paper: https://www.semanticscholar.org/paper/48d8418745a8847a2a3584f88d9d1f18f06957ad
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Davide Chicco
- Giuseppe Jurman
- Matthijs J. Warrens
- Stefano Marsili-Libelli
- John P. Norton
- Huapeng Li
- Xin Pan
- Anthony J. Jakeman
In Geography, Planning and Development
- Giuseppe Jurman
- Davide Chicco
- Patrick Hostert
- Tobias Kuemmerle
- Matthijs J. Warrens
- Joanne C. White
- Alexey A. Voinov
In Signal Processing
Citations in the corpus
- Characterising performance of environmental models (2013, background): “It has been criticized for inability to track the location of error (Kuhnert et al., 2006), and more recently Pontius and Millones (2010) have pronounced “death to Kappa” and declared the “birth of Quantity Disagreement and Allocation Disagreement for Accuracy”.”
- The Matthews Correlation Coefficient (MCC) is More Informative Than Cohen’s Kappa and Brier Score in Binary Classification Assessment (2021, methodology): “Pontius and Millones [70] criticized the Kappa statistic because it can generate values that do not make sense in remote sensing, and stated that Kappa coefficient’s statistically expected agreement can be irrelevant for the same domain.”
- Discrimination of crop types with TerraSAR-X-derived information (2015, background): “The total disagreement is the sum of 147 QD and AD (Baker et al, 2013; Pontius and Millones 2011).”
- Remote sensing of forest pest damage: a review and lessons learned from a Canadian perspective* (2016, background): “Other common metrics are overall accuracy (proportion of sampling units correctly classified) and the Kappa statistic, although there is a recent recommendation to replace the latter with two new indices, quantity disagreement and allocation disagreement (Pontius and Millones 2011).”
- Joint Deep Learning for land cover and land use classification (2019, methodology): “In addition, two recently proposed indices, including quantity disagreement and allocation disagreement, instead of the Kappa coefficient, were used to summarise comprehensively the confusion matrix of the classification results (Pontius and Millones, 2011).”
- A Review of the Application of Optical and Radar Remote Sensing Data Fusion to Land Use Mapping and Monitoring (2016, background): “More generally, assessing accuracy is challenging in the context of LUCC studies, where information on sampling designs, error matrices and accuracy measures, such as user’s, producer’s and overall accuracy, are not consistently and clearly reported [101,102], and there is still disagreement on reli”