Facing Imbalanced Data Recommendations for the Use of Performance Metrics

By László A. Jeni, Jeffrey F. Cohn, Fernando De la Torre. 2013

790 citations recorded by the source corpus.

Recognizing facial action units (AUs) is important for situation analysis and automated video annotation. Previous work has emphasized face tracking and registration and the choice of features classifiers. Relatively neglected is the effect of imbalanced data for action unit detection. While the machine learning community has become aware of the problem of skewed data for training classifiers, little attention has been paid to how skew may bias performance metrics. To address this question, we conducted experiments using both simulated classifiers and three major databases that differ in size, type of FACS coding, and degree of skew. We evaluated influence of skew on both threshold metrics (Accuracy, F-score, Cohen's kappa, and Krippendorf's alpha) and rank metrics (area under the receiver operating characteristic (ROC) curve and precision-recall curve). With exception of area under the ROC curve, all were attenuated by skewed distributions, in many cases, dramatically so. While ROC was unaffected by skew, precision-recall curves suggest that ROC may mask poor performance. Our findings suggest that skew is a critical factor in evaluating performance metrics. To avoid or minimize skew-biased estimates of performance, we recommend reporting skew-normalized scores along with the obtained ones.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistics and Probability, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistics and Probability

In Evaluation & Benchmarks

Citations in the corpus

Delving into Egocentric Actions (2015, neutral): “This can produce misleading results [13], as missing one instance in these “sparse” categories can impose a large penalty on average class accuracy.”
Machine Learning for the Diagnosis of Parkinson's Disease: A Review of Literature (2021, methodology): “In this case, evaluating machine learning models with other metrics including precision, recall and F-1 score is recommended (Jeni et al., 2013).”
Integrating Text and Image: Determining Multimodal Document Intent in Instagram Posts (2019, methodology): “der the ROC curve (AUC) (since AUC is more robust to class-skew), using macro-average across all classes (Jeni et al., 2013; Stager et al., 2006).”
A Review and Meta-Analysis of Multimodal Affect Detection Systems (2015, background): “Hence, the challenge is now to repudiate critics who think that affective systems will forever be resigned to the conﬁnes of the lab and will never make it into real-world real-world applications.”
Online Action Detection (2016, neutral): “This metric has one big disadvantage, though: it is sensitive to changes in the ratio of positive frames versus negative background frames (if the classiﬁers are not perfect), as discussed by Jeni et al. [35].”
Elastic sketch: adaptive and fast network-wide measurements (2018, background): “Unfortunately, the number of elephant flows is not known in advance and hard to predict [39].”
AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild (2017, methodology): “Table 9 shows accuracy, F1-score, Cohens kappa, Krippendorfs Alpha, area under the ROC curve (AUC), and area under the Precision-Recall curve (AUC-PR) on the test sets.”
Engagement detection in online learning: a review (2019, background): “Moreover, the issue of unbalanced data makes comparisons harder even further (Jeni et al. 2013).”