On the Convergence of a Family of Robust Losses for Stochastic Gradient Descent

By Bo Han, Ivor Wai-Hung Tsang, Ling Chen. 2016

22 citations recorded by the source corpus.

The convergence of Stochastic Gradient Descent SGD using convex loss functions has been widely studied. However, vanilla SGD methods using convex losses cannot perform well with noisy labels, which adversely affect the update of the primal variable in SGD methods. Unfortunately, noisy labels are ubiquitous in real world applications such as crowdsourcing. To handle noisy labels, in this paper, we present a family of robust losses for SGD methods. By employing our robust losses, SGD methods successfully reduce negative effects caused by noisy labels on each update of the primal variable. We not only reveal the convergence rate of SGD methods using robust losses, but also provide the robustness analysis on two representative robust losses. Comprehensive experimental results on six real-world datasets show that SGD methods using robust losses are obviously more robust than other baseline methods in most situations with fast convergence.

Subfields: Artificial Intelligence, Computational Mechanics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Philip S. Yu
2. Lichao Sun
3. Lingjuan Lyu
4. Qiang Yang
5. Han Yu
6. Ivor Wai-Hung Tsang
7. Xiaokui Xiao
8. Jun Zhao

In Computational Mechanics

1. Chris J. M. Yoon
2. Philip S. Yu
3. Qiang Yang
4. Ivor Wai-Hung Tsang
5. Clement Fung
6. Bo Han
7. Ling Chen
8. Ivan Beschastnikh

Citations in the corpus

• Mitigating Sybils in Federated Learning Poisoning (2018, background): “Training models with robust loss functions creates models that resist the presence of outliers [18].”
• The Limitations of Federated Learning in Sybil Settings (2020, background): “In the general ML setting, defenses rely on access to the training data [4] or access to the training process itself [7, 25, 45] to defend against adversaries.”
• Privacy-Preserving Stochastic Gradual Learning (2018, background): “The reason may be that, private curriculum in PRESTIGE introduces some noisy labels, and Gompertz loss can reduce robustness degeneration caused by these noisy labels on each update of the primal variable [50].”
• Privacy and Robustness in Federated Learning: Attacks and Defenses (2020, background): “Known defenses to poisoning attacks in a centralized setting, such as robust losses [126] and anomaly detection [57], assume control of the participants or explicit observation of the training data.”