Training Complex Models with Multi-Task Weak Supervision

By Alexander J. Ratner, Braden Hancock, Jared A. Dunnmon, Frederic Sala, Shreyash Pandey, Christopher R'e. 2018

227 citations recorded by the source corpus.

As machine learning models continue to increase in complexity, collecting large hand-labeled training sets has become one of the biggest roadblocks in practice. Instead, weaker forms of supervision that provide noisier but cheaper labels are often used. However, these weak supervision sources have diverse and unknown accuracies, may output correlated labels, and may label different tasks or apply at different levels of granularity. We propose a framework for integrating and modeling such weak supervision sources by viewing them as labeling different related sub-tasks of a problem, which we refer to as the multi-task weak supervision setting. We show that by solving a matrix completion-style problem, we can recover the accuracies of these multi-task sources given their dependency structure, but without any labeled data, leading to higher-quality supervision for training an end model. Theoretically, we show that the generalization error of models trained with this approach improves with the number of unlabeled data points, and characterize the scaling with respect to the task and dependency structures. On three fine-grained classification problems, we show that our approach leads to average gains of 20.2 points in accuracy over a traditional supervised approach, 6.8 points over a majority vote baseline, and 4.1 points over a previously proposed weak supervision method that models tasks separately.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

DataComp: In search of the next generation of multimodal datasets (2023, background): “The labeling functions could produce noisy and conflicting labels, so researchers have developed methods to aggregate noisy votes to produce the final training labels [110, 43, 125].”
Snorkel DryBell: A Case Study in Deploying Weak Supervision at Industrial Scale (2018, methodology): “It is also possible to directly plugin matrix factorization models of the kind recently used for denoising labeling functions [23] as TensorFlow model functions.”
A Survey on Programmatic Weak Supervision (2022, methodology): “MeTaL [ Ratner et al. , 2019 ] , instead, models the distribution via a Markov Network and recover the parameters via a matrix completion-style approach.”
Deep Learning--based Text Classification (2020, unknown): “99 93.2 - - 96.73 BERT-base [4,90] 95.63 93.5 96.04 61.6 98.08 70.58 BERT-large [4,90] 95.79 94.9 96.07 62.2 98.19 71.38 ALBERT [86] - 95.2 - - - - Multi-Task DNN [92] 83.2 95.6 - - - - Snorkel MeTaL [195] - 96.2 - - - - BERT Finetune + UDA [196] 95.8 96.5 62.88 97.95 62.92 RoBERTa (+additional data”
Ask Me Anything: A simple strategy for prompting language models (2022, background): “…a powerful framework that learns the accuracies and correlations of multiple noisy sources and aggregates them to produce weak labels for training data [Ratner et al., 2017, 2016, 2018, Varma et al., 2We draw inspiration from Wu et al. [2022] and focus on task-agnostic and scalable prompt-chains.”
Interactive Weak Supervision: Learning Useful Heuristics for Data Labeling (2020, methodology): “Prior data programming work of Ratner et al. (2016; 2019; 2020) with label models as in Eq.”
The Role of Massively Multi-Task and Weak Supervision in Software 2.0 (2019, methodology): “With respect to weak supervision, we build on the paradigm introduced in the Snorkel system [27, 29], and extended thereafter [3, 12, 32, 40, 41]; for a more comprehensive review of prior and related weak super-vision work, such as distant supervision [1, 15, 23] and crowdsourcing [4, 8, 9, 18,…”