Snuba: Automating Weak Supervision to Label Training Data

By Paroma Varma, Christopher R'e. 2018

161 citations recorded by the source corpus.

As deep learning models are applied to increasingly diverse problems, a key bottleneck is gathering enough high-quality training labels tailored to each task. Users therefore turn to weak supervision, relying on imperfect sources of labels like pattern matching and user-defined heuristics. Unfortunately, users have to design these sources for each task. This process can be time consuming and expensive: domain experts often perform repetitive steps like guessing optimal numerical thresholds and developing informative text patterns. To address these challenges, we present Snuba, a system to automatically generate heuristics using a small labeled dataset to assign training labels to a large, unlabeled dataset in the weak supervision setting. Snuba generates heuristics that each labels the subset of the data it is accurate for, and iteratively repeats this process until the heuristics together label a large portion of the unlabeled data. We develop a statistical measure that guarantees the iterative process will automatically terminate before it degrades training label quality. Snuba automatically generates heuristics in under five minutes and performs up to 9.74 F1 points better than the best known user-defined heuristics developed over many days. In collaborations with users at research labs, Stanford Hospital, and on open source datasets, Snuba outperforms other automated approaches like semi-supervised learning by up to 14.35 F1 points.

Subfields: Epidemiology, Pediatrics, Perinatology and Child Health

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Epidemiology

In Pediatrics, Perinatology and Child Health

Citations in the corpus

Grounding Predicates through Actions (2021, methodology): “dataset is not strictly unlabeled (we assume action labels are available), methods from this relatively new field could be applied to our framework in future work [32, 33].”
The Role of Massively Multi-Task and Weak Supervision in Software 2.0 (2019, methodology): “Another related approach aimed at enabling users to generate labeling functions without the need to directly write code is to use program synthesis techniques, combined with a small set of labeled data points, to automatically generate labeling functions [42].”
Challenges in Deploying Machine Learning: A Survey of Case Studies (2020, methodology): “Some of the most popular tools include Snorkel [93], Snuba [94] and cleanlab [95].”
A Survey on Programmatic Weak Supervision (2022, background): “For example, it often requires SMEs to explore a considerable amount of development data to generate ideas for LFs [ Varma and R´e, 2018; Galhotra et al. , 2021; Boecking et al. , 2021 ] .”
Interactive Weak Supervision: Learning Useful Heuristics for Data Labeling (2020, methodology): “For images, we can create a library of pre-trained object detectors as in (Chen et al., 2019), or in some applications combine primitives of geometric properties of the images (Varma & Ré, 2018).”