Interactive Weak Supervision: Learning Useful Heuristics for Data Labeling

By Benedikt Boecking, Willie Neiswanger, Eric P. Xing, Artur Dubrawski. 2020

80 citations recorded by the source corpus.

Obtaining large annotated datasets is critical for training successful machine learning models and it is often a bottleneck in practice. Weak supervision offers a promising alternative for producing labeled datasets without ground truth annotations by generating probabilistic labels using multiple noisy heuristics. This process can scale to large datasets and has demonstrated state of the art performance in diverse domains such as healthcare and e-commerce. One practical issue with learning from user-generated heuristics is that their creation requires creativity, foresight, and domain expertise from those who hand-craft them, a process which can be tedious and subjective. We develop the first framework for interactive weak supervision in which a method proposes heuristics and learns from user feedback given on each proposed heuristic. Our experiments demonstrate that only a small number of feedback iterations are needed to train models that achieve highly competitive test set performance without access to ground truth training labels. We conduct user studies, which show that users are able to effectively provide feedback on heuristics and that test set results track the performance of simulated oracles.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Adversarial Training for High-Stakes Reliability (2022, background): “Future work could add more tools (such as better suggestions for our human adversaries [50]) and study the relative effectiveness of the different tools, develop better training methods for human attackers, or more fully characterize properties of adversarial inputs to better understand our models…”
Data-centric Artificial Intelligence: A Survey (2023, unknown): “Labeling Crowdsourced labeling Collaboration Full [58, 130, 226] Semi-supervised labeling Collaboration Partial [46, 174, 294, 297] Active learning Collaboration Partial [55, 66, 192, 273] Data programming Collaboration Partial [27, 80] Data programming Collaboration Minimum [103, 190, 191, 277] Dis”
A Survey on Programmatic Weak Supervision (2022, methodology): “…the generation of LFs after given a seed supervision set, interactive generation approaches cast LF development as an interactive process where users are iteratively queried for feed-back used in discovering useful LFs from a large set of candidates [Galhotra et al. , 2021; Boecking et al. , 2021].”