Snorkel DryBell: A Case Study in Deploying Weak Supervision at Industrial Scale

By Stephen H. Bach, Daniel Rodríguez, Yintao Liu, Yintao Liu, Chong Luo, Haidong Shao, Cassandra Xia, Souvik Sen, Alexander J. Ratner, Braden Hancock and 4 more. 2018

155 citations recorded by the source corpus.

Labeling training data is one of the most costly bottlenecks in developing machine learning-based applications. We present a first-of-its-kind study showing how existing knowledge resources from across an organization can be used as weak supervision in order to bring development time and cost down by an order of magnitude, and introduce Snorkel DryBell, a new weak supervision management system for this setting. Snorkel DryBell builds on the Snorkel framework, extending it in three critical aspects: flexible, template-based ingestion of diverse organizational knowledge, cross-feature production serving, and scalable, sampling-free execution. On three classification tasks at Google, we find that Snorkel DryBell creates classifiers of comparable quality to ones trained with tens of thousands of hand-labeled examples, converts non-servable organizational resources to servable models for an average 52% performance improvement, and executes over millions of data points in tens of minutes.

Subfields: Artificial Intelligence, Information Systems, Management Science and Operations Research, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Management Science and Operations Research

In Natural Language Processing

Citations in the corpus

SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (2019, background): “(2019d,c) and Bach et al. (2018) get further improvements respectively via multi-task finetuning and using massive amounts of weak supervision.”
A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios (2020, methodology): “Instead of having a set of tasks, Rahimi et al. (2019) built an ensemble of language-specific NER models which are then weighted depending on the zero- or few-shot target language.”
Challenges in Deploying Machine Learning: A Survey of Case Studies (2020, methodology): “Some of the most popular tools include Snorkel [93], Snuba [94] and cleanlab [95].”
DataComp: In search of the next generation of multimodal datasets (2023, background): “[6] Stephen H Bach, Daniel Rodriguez, Yintao Liu, Chong Luo, Haidong Shao, Cassandra Xia, Souvik Sen, Alex Ratner, Braden Hancock, Houman Alborzi, Rahul Kuchhal, Christopher Ré, and Rob Malkin.”
Snorkel: rapid training data creation with weak supervision (2019, methodology): “Further empirical validation of these strong unlabeled scaling results can be found in follow-up work using Snorkel in a range of application domains, including aortic valve classification in MRI videos [17], industrial-scale content classification at Google [4], fine-grained named entity recognitio”
A Survey on Programmatic Weak Supervision (2022, methodology): “For example, in a product classiﬁcation task at Google, [Bach et al. , 2019] leverages existing semantic topic model to identify contents irrelevant to the category of products of interest.”
Slice-based Learning: A Programming Model for Residual Learning in Critical Data Slices (2019, background): “Weak supervision has been viewed as a new way to incorporate data of varying accuracy sources, including domain experts, crowd sourcing, data augmentations, and external knowledge bases [2, 5, 6, 11, 14, 21, 25, 29, 31].”
Learning from Rules Generalizing Labeled Exemplars (2020, background): “There is growing interest in learning from such efﬁcient, albiet noisy, supervision (Ratner et al., 2016; Pal & Balasubramanian, 2018; Bach et al., 2019; Sun et al., 2018; Kang et al., 2018).”
Preserving integrity in online social networks (2020, methodology): “Programmatic labeling of data can be a useful tool in this context [14].”
Interactive Weak Supervision: Learning Useful Heuristics for Data Labeling (2020, methodology): “These weak supervision approaches have been used on a wide variety of data types such as MRI sequences and unstructured text, and in various domains such as healthcare and e-commerce (Fries et al., 2019; Halpern et al., 2014; Bach et al., 2019; Ré et al., 2020).”