Data Programming: Creating Large Training Sets, Quickly

By Alexander J. Ratner, Christopher De Sa, Sen Wu, Daniel Selsam, Christopher R'e. 2016

780 citations recorded by the source corpus.

Large labeled training sets are the critical building blocks of supervised learning methods and are key enablers of deep learning techniques. For some applications, creating labeled training sets is the most time-consuming and expensive part of applying machine learning. We therefore propose a paradigm for the programmatic creation of training sets called data programming in which users express weak supervision strategies or domain heuristics as labeling functions, which are programs that label subsets of the data, but that are noisy and may conflict. We show that by explicitly representing this training set labeling process as a generative model, we can “denoise” the generated training set, and establish theoretically that we can recover the parameters of these generative models in a handful of settings. We then show how to modify a discriminative loss function to make it noise-aware, and demonstrate our method over a range of discriminative models including logistic regression and LSTMs. Experimentally, on the 2014 TAC-KBP Slot Filling challenge, we show that data programming would have led to a new winning score, and also show that applying data programming to an LSTM model leads to a TAC-KBP score almost 6 F1 points over a state-of-the-art LSTM baseline (and into second place in the competition). Additionally, in initial user studies we observed that data programming may be an easier way for non-experts to create machine learning models when training data is limited or unavailable.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Snorkel: Rapid Training Data Creation with Weak Supervision (2017, methodology): “Tradeoffs in Modeling of Sources: Snorkel learns the accuracies of weak supervision sources without access to ground truth using a generative model [38].”
Training Classifiers with Natural Language Explanations (2018, methodology): “We choose instead to use them as functions for weakly supervising the creation of a larger training set via data programming (Ratner et al., 2016).”
Teaching Autoregressive Language Models Complex Tasks By Demonstration (2021, methodology): “In this respect, it is similar to the concept of data programming [10], [11], an approach to creating training datasets with numerous imperfect but easy-to-generate ‘labeling functions’ that has proven popular and successful in industry.”
A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective (2018, background): “As data labeling at scale becomes more important especially for deep learning applications, data programming [126] has been proposed as a solution for generating large amounts of”
Label-Free Supervision of Neural Networks with Physics and Domain Knowledge (2016, background): “2010), and the recent work of (Ratner et al. 2016) has provided a fresh perspective on the idea of learning with labeling functions, rather than labels, in the form of Data Programming.”
Neurosymbolic Programming (2021, methodology): “A final category in this topic is programmatic supervision (Ratner et al., 2016; Zhan et al., 2020), which is closely related to the concepts of self-supervision (Hendrycks et al., 2019; Chen et al., 2020a) and data augmentation (Taylor and Nitschke, 2017).”
On the Accuracy of Influence Functions for Measuring Group Effects (2019, background): “It was collected via data programming, where users provide labeling functions (LFs)—instead of labels— that take in an unlabeled point and either abstain or output a heuristic label (Ratner et al., 2016).”
Data Lifecycle Challenges in Production Machine Learning (2018, methodology): “Weak supervision is used in Snorkel [57, 56] where hand labeling is avoided altogether, and workers can programmatically generate lower-quality labels, which are then denoised with generative models.”
Curriculum Loss: Robust Learning and Generalization against Label Corruption (2019, background): “For instance, noisy corrupted (wrong) labels may be resulted from annotating for similar objects [19], crawling images and labels from websites [7, 21] and creating training sets by program [18, 12].”
Attentive Semantic Video Generation Using Captions (2017, background): “Quantitative Results Data programming: Since our network can generate caption based videos, it can even be employed to artificially create a labeled dataset [14].”
Snorkel: Fast Training Set Generation for Information Extraction (2017, background): “Snorkel’s workflow is fundamentally different from traditional machine learning approaches, and is based on the newly proposed data programming paradigm [4].”
Learning from Rules Generalizing Labeled Exemplars (2020, methodology): “There is growing interest in learning from such efﬁcient, albiet noisy, supervision (Ratner et al., 2016; Pal & Balasubramanian, 2018; Bach et al., 2019; Sun et al., 2018; Kang et al., 2018).”
A clinical text classification paradigm using weak supervision and deep representation (2019, methodology): “According to the mean value theorem and Cauchy-Schwarz inequality [46], we can derive an upper bound for the difference between the loss function using true labels (i.e., L ( w )) and that using weak labels (i.e., L ϕ ( w )), i.e., where c is a constant value.”