Identifying Mislabeled Data using the Area Under the Margin Ranking

By Geoff Pleiss, Tianyi Zhang, Ethan R. Elenberg, Kilian Q. Weinberger. 2020

378 citations recorded by the source corpus.

Not all data in a typical training set help with generalization; some samples can be overly ambiguous or outrightly mislabeled. This paper introduces a new method to identify such samples and mitigate their impact when training neural networks. At the heart of our algorithm is the Area Under the Margin (AUM) statistic, which exploits differences in the training dynamics of clean and mislabeled samples. A simple procedure - adding an extra class populated with purposefully mislabeled indicator samples - learns a threshold that isolates mislabeled data based on this metric. This approach consistently improves upon prior work on synthetic and real-world datasets. On the WebVision50 classification task our method removes 17% of training data, yielding a 2.6% (absolute) improvement in test error. On CIFAR100 removing 13% of the data leads to a 1.2% drop in error.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Understanding Dataset Difficulty with V-Usable Information (2021, neutral): “As a notable exception, Zhang et al. (2020) proposed DIME, an information-theoretic measure to estimate a lower bound on the lowest possible (i.e., model-agnostic) 0-1 error.”
Continual Learning on Noisy Data Streams via Self-Purified Replay (2021, methodology): “Table 3 compares the purification performance with the state-of-the-art noise detection methods based on the training dynamics, including AUM [64] and INCV [11].”
Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics (2020, methodology): “, 2019), Shapely values (Ghorbani and Zou, 2019), and the area-under-margin metric (Pleiss et al., 2020) have all been used to identify mislabeled examples.”
Robust early-learning: Hindering the memorization of noisy labels (2021, background): “These settings are widely used in existing works (Ma et al., 2018; Thulasidasan et al., 2019; Pleiss et al., 2020; Wang et al., 2021a).”
Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamics (2022, background): “A simple yet widely-used approach to mitigate label noise is to remove the impacted data examples (Pleiss et al., 2020).”
Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards? (2021, background): “For example, Swayamdipta et al. (2020) and Pleiss et al. (2020) (Boyd-Graber and Börschinger, 2020), prioritizing examples most likely to reveal interesting distinctions between submitted systems.”
Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond (2021, methodology): “We hope the taxonomy can shed light on future improvements/extensions on explaining Table 2 List of interpretation algorithm publications Methods Publications (non-exhaustive) LIME and variants LIME [137], Anchors [138], SHAP [110], RISE [127], MAPLE [130] Global interpretation LIME-SP [137], NormLI”
Information-Theoretic Measures of Dataset Difficulty (2021, methodology): “It follows earlier work that used the training loss (Han et al., 2018; Arazo et al., 2019; Shen and Sanghavi, 2019), conﬁdence (Hovy et al., 2013), prediction variance (Chang et al., 2017), and area under the curve (Pleiss et al., 2020) to differentiate instances.”