Snorkel: rapid training data creation with weak supervision

By Alexander J. Ratner, Stephen H. Bach, Henry R. Ehrenberg, Jason Alan Fries, Sen Wu, Christopher R'e. 2019

277 citations recorded by the source corpus.

Labeling training data is increasingly the largest bottleneck in deploying machine learning systems. We present Snorkel, a first-of-its-kind system that enables users to train state-of-the-art models without hand labeling any training data. Instead, users write labeling functions that express arbitrary heuristics, which can have unknown accuracies and correlations. Snorkel denoises their outputs without access to ground truth by incorporating the first end-to-end implementation of our recently proposed machine learning paradigm, data programming. We present a flexible interface layer for writing labeling functions based on our experience over the past year collaborating with companies, agencies, and research laboratories. In a user study, subject matter experts build models 2.8×\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$2.8\times $$\end{document} faster and increase predictive performance an average 45.5%\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$45.5\%$$\end{document} versus seven hours of hand labeling. We study the modeling trade-offs in this new setting and propose an optimizer for automating trade-off decisions that gives up to 1.8×\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$1.8\times $$\end{document} speedup per pipeline execution. In two collaborations, with the US Department of Veterans Affairs and the US Food and Drug Administration, and on four open-source text and image data sets representative of other deployments, Snorkel provides 132%\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$132\%$$\end{document} average improvements to predictive performance over prior heuristic approaches and comes within an average 3.60%\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$3.60\%$$\end{document} of the predictive performance of large hand-curated training sets.

Subfields: Artificial Intelligence, Health Informatics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

Citations in the corpus

Advances, challenges and opportunities in creating data for trustworthy AI (2022, background): “One approach to reduce annotation costs is data programming 61–63 .”
A Survey on Programmatic Weak Supervision (2022, background): “Since the probabilistic training labels derived from the label model may still contain noise, [ Ratner et al. , 2017 ] suggests using a noise-aware loss as the training objective for the end model.”
Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision (2023, background): “Weak-to-strong learning is a special type of weakly supervised learning—a setting in which models are trained using unreliable labels (Bach et al., 2017; Ratner et al., 2017; Guo et al., 2018).”
A Survey of Deep Learning for Scientific Discovery (2020, methodology): “Other useful augmentation strategies include TANDA [188] which learns a model to compose data augmentations, the related randaugment [38], choosing a random subset of diﬀerent possible augmentations, population based augmentation [85] which randomly searches over diﬀerent augmentation policies,…”
Slice-based Learning: A Programming Model for Residual Learning in Critical Data Slices (2019, methodology): “We reproduce the DP [25] setup to obtain overall scores of F1= 41 .”
Training Complex Models with Multi-Task Weak Supervision (2018, methodology): “These approaches of effectively programming machine learning models have recently been formalized by the following pipeline (Ratner et al. 2016; 2018): First, users provide one or more weak supervision sources, which are applied to unlabeled data to generate a set of noisy labels.”
In-N-Out: Pre-Training and Self-Training using Auxiliary Information for Out-of-Distribution Robustness (2020, background): “Similarly, weak supervision works assume access to weak labels correlated with the true label [37, 36].”
Ask Me Anything: A simple strategy for prompting language models (2022, methodology): “3: Learn Ĝ = (V, Ê) via structure learning on DP (Algorithm 1 in Varma et al. [2019]).”
Data collection and quality challenges in deep learning: a data-centric AI perspective (2021, background): “Snorkel [10,122,123] is the seminal system for data pro-”