Learning with Bad Training Data via Iterative Trimmed Loss Minimization

By Yanyao Shen, Sujay Sanghavi. 2018

297 citations recorded by the source corpus.

Subfields: Optimization & Training, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

In Safety, Robustness & Interpretability

Citations in the corpus

Understanding Dataset Difficulty with V-Usable Information (2021, neutral): “Other approaches consider training loss (Han et al., 2018; Arazo et al., 2019; Shen & Sanghavi, 2019), confidence (Hovy et al., 2013), prediction variance (Chang et al., 2017), and area under the curve (Pleiss et al., 2020).”
Advances and Open Problems in Federated Learning (2019, methodology): “ently rely on “data sanitization” methods [123], which aim to remove poisoned or otherwise anomalous 65 data. More recent work has developed improved data sanitization methods using robust statistics [368, 355, 387, 133], which often have the beneﬁt of being provably robust to small numbers of outli”
Can You Really Backdoor Federated Learning? (2019, background): “Existing approaches against backdoor attacks (28; 20; 31; 13; 32; 26) either require a careful examination of the training data or full control of the training process at the server, which may not apply in the federated learning case.”
Continual Learning on Noisy Data Streams via Self-Purified Replay (2021, background): “There have been multiple lines of work to learn amidst noisy labels such as loss regularization [84, 96, 31], data re-weighting [68, 72], label cleaning [64, 42, 61], and”
Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics (2020, methodology): “Other approaches have used training loss (Han et al., 2018; Arazo et al., 2019; Shen and Sanghavi, 2019), confidence (Hovy et al.”
DORO: Distributional and Outlier Robust Optimization (2021, unknown): “In Section 3 we used a simple version of iterative trimming (Shen & Sanghavi, 2019) to remove outliers from the training set.”
Robust Learning by Self-Transition for Handling Noisy Labels (2020, methodology): “• ITLM: Because it iterates the training process multiple times as well, the total number of training rounds was set to be 5.”
Learning from Rules Generalizing Labeled Exemplars (2020, methodology): “Others in this category include: Shen & Sanghavi (2019)’s method of iteratively selecting examples with smallest loss, and Veit et al. (2017)’s method of learning a separate network to transform noisy labels to cleaned ones which are used to impose a cross-entropy loss on P θ ( y | x ) .”
Information-Theoretic Measures of Dataset Difficulty (2021, methodology): “It follows earlier work that used the training loss (Han et al., 2018; Arazo et al., 2019; Shen and Sanghavi, 2019), conﬁdence (Hovy et al., 2013), prediction variance (Chang et al., 2017), and area under the curve (Pleiss et al., 2020) to differentiate instances.”
Identifying Mislabeled Data using the Area Under the Margin Ranking (2020, background): “The vast majority of approaches use training loss as a proxy for label quality, as mislabeled samples tend to have a higher loss at early epochs (Zhang et al., 2017; Shen & Sanghavi, 2019).”
Anti-Backdoor Learning: Training Clean Models on Poisoned Data (2021, background): “An early work [34] found that DNNs are more accurate on clean samples in an early training stage, while we find that the backdoor attacks studied in [34] are only limited to simple BadNets attacks.”
Choosing the Sample with Lowest Loss makes SGD Robust (2020, background): “Recently, [5, 6, 32, 20, 18] have proposed robust learning algorithms for linear regression which require the computation of gradient over the entire dataset which may be computationally intractable for large datasets.”
Estimating Training Data Influence by Tracking Gradient Descent (2020, background): “[15]) that don’t use the notion of training data influence/attribution.”
A Noise-Robust Framework for Automatic Segmentation of COVID-19 Pneumonia Lesions From CT Images (2020, methodology): “First, we propose an adaptive teacher by suppressing the contribution of S to T (i.e., EMA) when S has a poor performance with large training loss values that may be caused by noisy labels [31].”
Learning from data in the mixed adversarial non-adversarial case: Finding the helpers and ignoring the trolls (2022, background): “…(Sukhbaatar et al., 2014; Xiao et al., 2015)), robust loss functions (e.g., (Ghosh et al., 2017; Liu and Guo, 2020)), robust regularization (e.g., (Jenni and Favaro, 2018; Goodfellow et al., 2014)) and sample selection methods (e.g., (Malach and Shalev-Shwartz, 2017; Shen and Sanghavi, 2019)).”