Simple and Effective Regularization Methods for Training on Noisily Labeled Data with Generalization Guarantee

By Wei Hu, Zhiyuan Li, Dingli Yu. 2019

143 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descent (2020, background): “These bounds share some similarities with the bounds derived by Arora et al. (2019), Liu et al. (2019) and Hu et al. (2019), where in these works, the bounds were derived for supervised learning settings, and in some cases for two-layer RELU neural networks.”
Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics (2020, background): “Recent studies have shown that over-parameterized neural networks can ﬁt the incorrect labels blindly (Zhang et al., 2017), which might hurt their generalization ability (Hu et al., 2020).”
Identifying Mislabeled Data using the Area Under the Margin Ranking (2020, background): “Li et al. (2019) and Hu et al. (2019) develop theoretical guarantees in the noisy-label setting for early stopping and other regulariza-tions.”
Robust early-learning: Hindering the memorization of noisy labels (2021, methodology): “It would be interesting to explore the potential characteristics of the updated parameters such as the distance to initial parameters (Li et al., 2020b; Hu et al., 2020) and the mutual information with the vector of all training labels given inputs (Harutyunyan et al.”
CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information (2020, methodology): “…the network capacity ( i.e. , adding layers and neurons) and applying gradient-ascent to the log-likelihood L ( θ ) , we can obtain far more accurate approximation q θ ( y | x ) to p ( y | x ) , thanks to the high expressiveness of neural networks (Hu et al., 2019; Oymak & Soltanolkotabi, 2019).”
Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networks (2019, result): “Our regularizer penalizes W , i.e. the distance from initialization, similar as in (Hu et al., 2019).”
Noise against noise: stochastic label noise helps combat inherent label noise (2021, methodology): “• DivideMix (Li et al., 2020 Speciﬁcally, we use models trained with SLN as the initialization of these methods.”
Improving Generalization by Controlling Label-Noise Information in Neural Network Weights (2020, background): “For isotropic Gaussians with equal covariances, the KL divergence reduces to ‖θ − θ0‖2, which was studied by Hu et al. (2020) as a regularization to achieve robustness to label-noise.”