Snorkel: Rapid Training Data Creation with Weak Supervision

By Alexander J. Ratner, Stephen H. Bach, Henry R. Ehrenberg, Jason Alan Fries, Sen Wu, Christopher R'e. 2017

1,174 citations recorded by the source corpus.

Labeling training data is increasingly the largest bottleneck in deploying machine learning systems. We present Snorkel, a first-of-its-kind system that enables users to train state-of- the-art models without hand labeling any training data. Instead, users write labeling functions that express arbitrary heuristics, which can have unknown accuracies and correlations. Snorkel denoises their outputs without access to ground truth by incorporating the first end-to-end implementation of our recently proposed machine learning paradigm, data programming. We present a flexible interface layer for writing labeling functions based on our experience over the past year collaborating with companies, agencies, and research labs. In a user study, subject matter experts build models 2.8× faster and increase predictive performance an average 45.5% versus seven hours of hand labeling. We study the modeling tradeoffs in this new setting and propose an optimizer for automating tradeoff decisions that gives up to 1.8× speedup per pipeline execution. In two collaborations, with the U.S. Department of Veterans Affairs and the U.S. Food and Drug Administration, and on four open-source text and image data sets representative of other deployments, Snorkel provides 132% average improvements to predictive performance over prior heuristic approaches and comes within an average 3.60% of the predictive performance of large hand-curated training sets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Evaluation & Benchmarks

Citations in the corpus

XLNet: Generalized Autoregressive Pretraining for Language Understanding (2019, background): “1 Multi-task ensembles on test (from leaderboard as of June 19, 2019) Snorkel∗ [29] 87.”
Learning from Dialogue after Deployment: Feed Yourself, Chatbot! (2019, positive): “, 2019) and relation extraction have observed similar results (Bunescu and Mooney, 2007; Hancock et al., 2018; Ratner et al., 2017).”
Training Classifiers with Natural Language Explanations (2018, methodology): “Because this task requires specialized domain expertise, we obtained explanations by having someone unfamiliar with BabbleLabble translate from Python to natural language labeling functions from an existing publication that explored applying weak supervision to this task (Ratner et al., 2018).”
Teaching Autoregressive Language Models Complex Tasks By Demonstration (2021, methodology): “In this respect, it is similar to the concept of data programming [10], [11], an approach to creating training datasets with numerous imperfect but easy-to-generate ‘labeling functions’ that has proven popular and successful in industry.”
Measuring Data (2022, methodology): “Recent tools in this vein include CleanLab (2022), DataLab (Xiao et al., 2022) and Snorkel (Ratner et al., 2017), which provide solutions for analyzing and curating quality ML datasets, but are limited in scope to absent methods for systematic evaluation of the data.”
A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective (2018, background): “Snorkel [127], [155] is the most recent system for data programming.”
Bootstrapping Conversational Agents With Weak Supervision (2018, background): “…• Guidance on creating effective queries: different from the conventional usage of a search engine–i.e., ﬁnding the single best answer, the use of search in a data programming context should target high coverage (Ratner et al. 2017) without over-sacriﬁcing the precision in retrieving examples.”
LIREx: Augmenting Language Inference with Relevant Explanation (2020, methodology): “Taking insights from the field of weak supervision (Fries et al. 2017; Bach et al. 2017; Ratner et al. 2020) where weakly labeled data is used for training models, we treat the selected explanations as weakly selected instances.”
An Empirical Analysis of Backward Compatibility in Machine Learning Systems (2020, methodology): “Due to high financial cost and time, the team may resort to noisier approaches to create this large dataset, such as crowdsourcing labels [23] or using weak supervision [34], a popular alternative due to its automation opportunities.”
A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios (2020, background): “encompass also other ideas like reg-ex rules or simple programming functions (Ratner et al., 2017; Zheng et al., 2019; Adelani et al., 2020; Hedderich et al., 2020; Lison et al., 2020; Ren et al., 2020; Karamanolakis et al., 2021).”
Fonduer: Knowledge Base Construction from Richly Formatted Data (2017, positive): “To perform data programming in Fonduer, we rely on a dataprogramming engine, Snorkel [32].”
Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision (2023, background): “Weak-to-strong learning is a special type of weakly supervised learning—a setting in which models are trained using unreliable labels (Bach et al., 2017; Ratner et al., 2017; Guo et al., 2018).”
The Role of Massively Multi-Task and Weak Supervision in Software 2.0 (2019, methodology): “In response to this, over the past three years we have developed Snorkel [27] 1 , a system for programmatically creating and modeling training data for tasks over arbitrary data types such as text, images, time series, financial data, and many others.”
Generative Multi-Agent Behavioral Cloning (2018, background): “Players are ordered based on their relative positions, similar to the role assignment in Lucey et al. [2013]. Overall, there are 107,146 training and 13,845 test examples.”
Osprey: Weak Supervision of Imbalanced Extraction Problems without Code (2019, methodology): “Another significant advantage of a weakly-supervised approach is that upon a data change necessitating a model In the basic setup of Snorkel [17], LFs are usually described as being developed from scratch, on demand, by a domain expert.”
In-N-Out: Pre-Training and Self-Training using Auxiliary Information for Out-of-Distribution Robustness (2020, background): “Similarly, weak supervision works assume access to weak labels correlated with the true label [36, 35].”