Decoupling "when to update" from "how to update"

By Eran Malach, Shai Shalev-Shwartz. 2017

655 citations recorded by the source corpus.

Deep learning requires data. A useful approach to obtain data is to be creative and mine data from various sources, that were created for different purposes. Unfortunately, this approach often leads to noisy labels. In this paper, we propose a meta algorithm for tackling the noisy labels problem. The key idea is to decouple "when to update" from "how to update". We demonstrate the effectiveness of our algorithm by mining data for gender classification by combining the Labeled Faces in the Wild (LFW) face recognition dataset with a textual genderizing service, which leads to a noisy dataset. While our approach is very simple to implement, it leads to state-of-the-art results. We analyze some convergence properties of the proposed algorithm.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Co-teaching: Robust training of deep neural networks with extremely noisy labels (2018, methodology): “Decoupling [26], which updates the parameters only using the samples which have different prediction from two classifiers; and (v).”
A Comprehensive Survey on Poisoning Attacks and Countermeasures in Machine Learning (2022, methodology): “[76] proposed a method to train a model dynamically.”
Model-Agnostic Approaches To Handling Noisy Labels When Training Sound Event Classifiers (2019, background): “The state-of-the-art is based on selecting the clean data instances in the train set in order to train the network satisfactorily [9, 10, 11].”
Normalized Loss Functions for Deep Learning with Noisy Labels (2020, background): “Decoupling training strategy (Malach & Shalev-Shwartz, 2017) trains two networks simultaneously, and parameters are updated when their predictions disagree.”
SELFIE: Refurbishing Unclean Samples for Robust Deep Learning (2019, background): “Decouple (Malach & Shalev-Shwartz, 2017) performs the decoupling of when to update from how to update.”
Robust Learning by Self-Transition for Handling Noisy Labels (2020, methodology): “Decouple [25] maintains two DNNs simultaneously and updates the models by only using the samples with different label predictions from these two DNNs.”
Robust Inference via Generative Classifiers for Handling Noisy Labels (2019, methodology): “We also consider the following methods that require to train multiple networks, i.e., an ensemble of classiﬁers or a meta-learning model: Decoupling (Malach & Shalev-Shwartz, 2017), MentorNet (Jiang et al., 2018) and Co-teaching (Han et al., 2018b).”
Simple and Effective Regularization Methods for Training on Noisily Labeled Data with Generalization Guarantee (2019, background): “, 2017), and training two networks simultaneously (Malach and Shalev-Shwartz, 2017; Han et al., 2018; Yu et al., 2019).”
Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks (2019, background): “rved to be highly resilient to label noise and generalize well despite large corruption [44]. In addition to early stopping, several heuristics have been proposed to specically deal with label noise [27,31,37,42,47,57]. See also [23,38,43,48] for additional work on dealing with label noise in classi”
Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey (2019, background): “Multiple Classifiers: Consistency of networks [89], co-teaching [90, 91, 92, 93]”
Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels (2019, methodology): “An alternative approach is training on selected or weighted samples, e.g., Decoupling (Malach & Shalev-Shwartz, 2017), MentorNet (Jiang et al., 2018), gradient-based reweighting (Ren et al., 2018) and Co-teaching (Han et al., 2018).”
O2U-Net: A Simple Noisy Label Detection Approach for Deep Neural Networks (2019, neutral): “Another similar work is proposed in [11].”