Socratic Learning: Augmenting Generative Models to Incorporate Latent Subsets in Training Data

By Paroma Varma, Bryan D. He, Dan Iter, Peng Xu, Rose Yu, Christopher De Sa, Christopher R'e. 2016

26 citations recorded by the source corpus.

A challenge in training discriminative models like neural networks is obtaining enough labeled training data. Recent approaches use generative models to combine weak supervision sources, like user-defined heuristics or knowledge bases, to label training data. Prior work has explored learning accuracies for these sources even without ground truth labels, but they assume that a single accuracy parameter is sufficient to model the behavior of these sources over the entire training set. In particular, they fail to model latent subsets in the training data in which the supervision sources perform differently than on average. We present Socratic learning, a paradigm that uses feedback from a corresponding discriminative model to automatically identify these subsets and augments the structure of the generative model accordingly. Experimentally, we show that without any ground truth labels, the augmented generative model reduces error by up to 56.06% for a relation extraction task compared to a state-of-the-art weak supervision technique that utilizes generative models.

Subfields: Artificial Intelligence, Education, Generative Models, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Education

Percy Liang

In Generative Models

In Representation & Self-Supervised Learning

Citations in the corpus

Training Classifiers with Natural Language Explanations (2018, background): “Various extensions have focused on aggregating a variety of supervision sources by learning generative models from noisy labels (Alfonseca et al., 2012; Takamatsu et al., 2012; Roth and Klakow, 2013; Ratner et al., 2016; Varma et al., 2017).”
Inferring Generative Model Structure with Static Analysis (2017, background): “Recently, generative models have been used to create training labels from various weak supervision sources, such as heuristics or knowledge bases, by modeling the true class label as a latent variable [1, 2, 27, 31, 36, 37].”
Zero-Shot Task Transfer (2019, methodology): “Many vision [30] and non-vision machine learning applications [32], [38] encode such crowd wisdom in their learning methods.”
Interactive Weak Supervision: Learning Useful Heuristics for Data Labeling (2020, background): “, 2019), the use of noisy pairwise labeling functions (Boecking & Dubrawski, 2019), addressing latent subsets in the data (Varma et al., 2016), and fast model iteration via the use of pre-trained embeddings (Chen et al.”