Learning the Structure of Generative Models without Labeled Data

By Stephen H. Bach, Bryan D. He, Alexander J. Ratner, Christopher R'e. 2017

182 citations recorded by the source corpus.

Curating labeled training data has become the primary bottleneck in machine learning. Recent frameworks address this bottleneck with generative models to synthesize labels at scale from weak supervision sources. The generative model’s dependency structure directly affects the quality of the estimated labels, but selecting a structure automatically without any labeled data is a distinct challenge. We propose a structure estimation method that maximizes the ℓ1-regularized marginal pseudolikelihood of the observed data. Our analysis shows that the amount of unlabeled data required to identify the true structure scales sublinearly in the number of possible dependencies for a broad class of models. Simulations show that our method is 100× faster than a maximum likelihood approach and selects 1/4 as many extraneous dependencies. We also show that our method provides an average of 1.5 F1 points of improvement over existing, user-developed information extraction applications on real-world data such as PubMed journal abstracts.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

Snorkel: Rapid Training Data Creation with Weak Supervision (2017, methodology): “We optimize this objective by interleaving stochastic gradient descent steps with Gibbs sampling ones, similar to contrastive divergence [21]; for more details, see [5, 38].”
A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective (2018, background): “Theoretical analysis [3] shows that, if the labeling functions are reasonably accurate, then the predictions made by the generative model becomes arbitrarily close to the true labels.”
Snorkel DryBell: A Case Study in Deploying Weak Supervision at Industrial Scale (2018, background): “, simple heuristic rules and pattern matching scripts written from scratch by users, which may have diverse accuracies and correlations [3, 29, 30].”
A Survey on Programmatic Weak Supervision (2022, methodology): “However, manually specifying the dependency structure would bring extra burden to practitioners; to reduce human efforts, researches have attempted to learn the dependency structure automatically [Bach et al. , 2017; Varma et al. , 2017; Varma et al. , 2019a].”
The Role of Massively Multi-Task and Weak Supervision in Software 2.0 (2019, methodology): “With respect to weak supervision, we build on the paradigm introduced in the Snorkel system [27, 29], and extended thereafter [3, 12, 32, 40, 41]; for a more comprehensive review of prior and related weak supervision work, such as distant supervision [1, 15, 23] and crowdsourcing [4, 8, 9, 18, 45], ”
Weakly supervised classification of rare aortic valve malformations using unlabeled cardiac MRI sequences (2018, background): “This approach leverages unlabeled data to 37 estimate the unobserved accuracies of labeling sources as well as infer complex 38 statistical dependencies among labeling functions [12,13].”
Bootstrapping Conversational Agents With Weak Supervision (2018, methodology): “We additionally learn dependencies between labeling functions through a structure learning process given by (Bach et al. 2017), by modeling the distribution in question as a factor graph.”
Generating Multi-Agent Trajectories using Programmatic Weak Supervision (2018, background): “Instead, our work is inspired by recent advances in weak supervision settings known as data programming , in which multiple weak and noisy label sources called labeling functions can be leveraged to learn the underlying structure of large unlabeled datasets (Ratner et al., 2018; Bach et al., 2017).”
Osprey: Weak Supervision of Imbalanced Extraction Problems without Code (2019, methodology): “Recent work has focused on building end-to-end systems allowing non-experts to create and manage multiple sources of weak supervision that may have diverse accuracies, coverages, and correlations [1, 18].”
Snorkel: rapid training data creation with weak supervision (2019, methodology): “Using the recently proposed approach of data programming [5,43], we model the true class label for a data point as a latent variable in a probabilistic model.”
Training Complex Models with Multi-Task Weak Supervision (2018, methodology): “Our work builds on and extends various settings studied in machine learning: Weak Supervision: We draw motivation from recent work which models and integrates weak supervision using generative models [30; 29; 2] and other methods [13; 19].”
Snuba: Automating Weak Supervision to Label Training Data (2018, methodology): “Over the past few years, we have been part of the broader effort to enhance methods based on user-defined heuristics to extend their applicability to text, image, and video data for tasks in computer vision, medical imaging, bioinformatics and knowledge base construction [40, 4, 52].”
Generative Multi-Agent Behavioral Cloning (2018, methodology): “nated multi-agent trajectories. An interesting direction is to incorporate multiple labeling functions, each viewed as noisy realizations of true macro-intents, similar to (Ratner et al., 2016; 2018; Bach et al., 2017). ACKNOWLEDGMENTS This research is supported in part by NSF #1564330, NSF #1637598”
LIREx: Augmenting Language Inference with Relevant Explanation (2020, methodology): “Taking insights from the field of weak supervision (Fries et al. 2017; Bach et al. 2017; Ratner et al. 2020) where weakly labeled data is used for training models, we treat the selected explanations as weakly selected instances.”
Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision (2023, background): “Weak-to-strong learning is a special type of weakly supervised learning—a setting in which models are trained using unreliable labels (Bach et al., 2017; Ratner et al., 2017; Guo et al., 2018).”
Snorkel: Fast Training Set Generation for Information Extraction (2017, methodology): “In Snorkel, we can also automatically learn dependencies that may exist between the labeling functions, further improving the accuracy of our model [1].”
Inferring Generative Model Structure with Static Analysis (2017, methodology): “Figure 4: Simulation demonstrating improved generative model accuracy with Coral compared to structure learning [2] and Coral.”
Zero-Shot Task Transfer (2019, background): “Structure learning [4] also exploits the use of distant supervision signals for generating labels.”
Socratic Learning: Augmenting Generative Models to Incorporate Latent Subsets in Training Data (2016, background): “Even without ground truth data, generative models have been successful in learning accuracies for [19] and dependencies among [2] the diﬀerent weak supervision sources.”