Without quality presence–absence data, discrimination metrics such as TSS can be misleading measures of model performance

By Boris Leroy, Robin Delsol, B. Hugueny, C. Meynard, Chéima Barhoumi, Morgane Barbet-Massin, C. Bellard. Journal of Biogeography, 2018

377 citations recorded by the source corpus.

The discriminating capacity (i.e. ability to correctly classify presences and absences) of species distribution models (SDMs) is commonly evaluated with metrics such as the area under the receiving operating characteristic curve (AUC), the Kappa statistic and the true skill statistic (TSS). AUC and Kappa have been repeatedly criticized, but TSS has fared relatively well since its introduction, mainly because it has been considered as independent of prevalence. In addition, discrimination metrics have been contested because they should be calculated on presence–absence data, but are often used on presence‐only or presence‐background data. Here, we investigate TSS and an alternative set of metrics—similarity indices, also known as F‐measures. We first show that even in ideal conditions (i.e. perfectly random presence–absence sampling), TSS can be misleading because of its dependence on prevalence, whereas similarity/F‐measures provide adequate estimations of model discrimination capacity. Second, we show that in real‐world situations where sample prevalence is different from true species prevalence (i.e. biased sampling or presence‐pseudoabsence), no discrimination capacity metric provides adequate estimation of model discrimination capacity, including metrics specifically designed for modelling with presence‐pseudoabsence data. Our conclusions are twofold. First, they unequivocally impel SDM users to understand the potential shortcomings of discrimination metrics when quality presence–absence data are lacking, and we recommend obtaining such data. Second, in the specific case of virtual species, which are increasingly used to develop and test SDM methodologies, we strongly recommend the use of similarity/F‐measures, which were not biased by prevalence, contrary to TSS.

Subfields: Ecological Modeling, Ecology, Genetics

View Paper: https://www.semanticscholar.org/paper/d04dfb895ef39bb0d56f920b09bea4e69c2eb7c9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Ecological Modeling

In Ecology

Citations in the corpus

The area under the precision‐recall curve as a performance metric for rare binary events (2019, background): “True prevalence is difficult to estimate from presence–absence data because of issues including sampling biases and imperfect detection; when absence data are unavailable, true prevalence is simply unknown (Guillera- Arroita et al., 2015; Leroy et al., 2018).”
Two alternative evaluation metrics to replace the true skill statistic in the assessment of species distribution models (2019, methodology): “2017) and the degree of confidence in absence data (Leroy et al. 2018) which is influenced by species mobility (Jaberg and Guisan 2001) and a multitude of other factors.”