Snorkel MeTaL: Weak Supervision for Multi-Task Learning

By Alexander J. Ratner, Braden Hancock, Jared A. Dunnmon, Roger E. Goldman, Christopher R'e. 2018

61 citations recorded by the source corpus.

Many real-world machine learning problems are challenging to tackle for two reasons: (i) they involve multiple sub-tasks at different levels of granularity; and (ii) they require large volumes of labeled training data. We propose Snorkel MeTaL, an end-to-end system for multi-task learning that leverages weak supervision provided at multiple levels of granularity by domain expert users. In MeTaL, a user specifies a problem consisting of multiple, hierarchically-related sub-tasks---for example, classifying a document at multiple levels of granularity---and then provides labeling functions for each sub-task as weak supervision. MeTaL learns a re-weighted model of these labeling functions, and uses the combined signal to train a hierarchical multi-task network which is automatically compiled from the structure of the sub-tasks. Using MeTaL on a radiology report triage task and a fine-grained news classification task, we achieve average gains of 11.2 accuracy points over a baseline supervised approach and 9.5 accuracy points over the predictions of the user-provided labeling functions.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Yi Tay
2. Mostafa Dehghani
3. Sebastian Ruder
4. Mohit Bansal
5. Yoshua Bengio
6. Percy Liang
7. Zhe Gan
8. Quoc V. Le

Citations in the corpus

• A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective (2018, background): “More recently, Snorkel has been extended to solve massivelymulti-task learning [130].”
• Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks (2021, background): “Specifically, when target datasets have limited training data, multi-task learning improves the performance compared to individually trained models (Liu et al., 2019a; Ratner et al., 2018).”
• CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT (2020, methodology): “A method of weak supervision known as data programming (Ratner et al., 2018) has seen successful application to medical report labeling: in this method, users write heuristic labelling functions that programmatically label training data.”
• Multi-task learning for natural language processing in the 2020s: where are we going? (2020, background): “While this work has focused on small task sets and relatively small combinations, others have shown the benefit of having many auxiliary tasks to boost MTL performance (Ruder, 2017; Ratner et al., 2018; Liu et al., 2019a).”
• The Role of Massively Multi-Task and Weak Supervision in Software 2.0 (2019, methodology): “…building on top of the basic labeling function abstraction used in Snorkel; second, by extending initial work on a multi-task version of Snorkel [31, 33] to a newly-proposed massively multi-task setting; and finally, by focusing on new modes of deploying production models directly from this…”