Fidelity-Weighted Learning

By Mostafa Dehghani, Arash Mehrjou, Stephan Gouws, Jaap Kamps, Bernhard Schölkopf. 2017

78 citations recorded by the source corpus.

Training deep neural networks requires many training samples, but in practice training labels are expensive to obtain and may be of varying quality, as some may be from trusted expert labelers while others might be from heuristics or other sources of weak supervision such as crowd-sourcing. This creates a fundamental quality versus-quantity trade-off in the learning process. Do we learn from the small amount of high-quality data or the potentially large amount of weakly-labeled data? We argue that if the learner could somehow know and take the label-quality into account when learning the data representation, we could get the best of both worlds. To this end, we propose "fidelity-weighted learning" (FWL), a semi-supervised student-teacher approach for training deep neural networks using weakly-labeled data. FWL modulates the parameter updates to a student network (trained on the task we care about) on a per-sample basis according to the posterior confidence of its label-quality estimated by a teacher (who has access to the high-quality labels). Both student and teacher are learned from the data. We evaluate FWL on two tasks in information retrieval and natural language processing where we outperform state-of-the-art alternative semi-supervised methods, indicating that our approach makes better use of strong and weak labels, and leads to better task-dependent data representations.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Pretrained Transformers for Text Ranking: BERT and Beyond (2020, neutral): “3124 [Dehghani et al., 2018].”
Zero-shot Knowledge Transfer via Adversarial Belief Matching (2019, background): “…with competing models such as that of Kimura et al. (2018) where authors must include hand designed losses like carbon copy memory replay (which freezes some pseudo samples in time), or ﬁdelity (Dehghani et al., 2017). x adv ← x adv − ξ ∂ L CE ( y A , j ) ∂ x adv save: y A , y B end end end end”
Few-shot learning of neural networks from scratch by pseudo example optimization (2018, methodology): “For this purpose, we introduce the idea of fidelity weighting [6], which adaptively weighs training examples based on uncertainty of predictions obtained from the reference model.”
MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels (2017, methodology): “In parallel to CL, the sample weighting schemes were also studied in (Lin et al., 2017a; Wang et al., 2017; Fan et al., 2018; Dehghani et al., 2018).”
Interactive Weak Supervision: Learning Useful Heuristics for Data Labeling (2020, background): “Weak supervision provides a promising alternative, reducing the need for humans to hand label large datasets to train ML models (Riedel et al., 2010; Hoffmann et al., 2011; Ratner et al., 2016; Dehghani et al., 2018).”
Cross-Domain Modeling of Sentence-Level Evidence for Document Retrieval (2019, methodology): “Prior to our work, the meta-analysis by Yang et al. (2019a), which analyzed over 100 papers up until early 2019, 4 put the best neural model at 0.3124 (Dehghani et al., 2018).”
Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation (2020, background): “Subsequently they demonstrate the success of a semi-supervised student-teacher approach for an information retrieval task using weakly labelled data where the teacher has access to the high quality labels [9].”
Meta-Weight-Net: Learning an Explicit Mapping For Sample Weighting (2019, methodology): “Typical methods along this line include FWL [51], learning to teach [52, 32] and MentorNet [21] methods, whose weight functions are designed as a Bayesian function approximator, a DNN with attention mechanism, a bidirectional LSTM network, respectively.”
Rethinking Class-Balanced Methods for Long-Tailed Visual Recognition From a Domain Adaptation Perspective (2020, unknown): “Similar schemes are used in learning from noisy data [29, 9, 52].”
Training Curricula for Open Domain Answer Re-Ranking (2020, unknown): “ted by Ferro et al. show that a curriculum learning strategy gives only a limited boost to the ranking performance of an ensemble of decision trees. Similar to our approach, Fidelityweighted learning [8] applies weights to training samples for learning ranking models. However, this approach focuses ”
Imitation networks: Few-shot learning of neural networks from scratch (2018, methodology): “For this purpose, we introduce fidelity weighting [Dehghani et al., 2017], which adaptively weighs training examples based on uncertainty of predictions obtained from the reference model With the introduction of fidelity weighting, the imitation loss Eq.”
Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey (2019, background): “Using data with both clean and noisy labels: iteratively correct [58], correct for fine-tune [59] c.”