Co-training Improves Prompt-based Learning for Large Language Models

By Hunter Lang, Monica Agrawal, Yoon Kim, David A. Sontag. 2022

47 citations recorded by the source corpus.

We demonstrate that co-training (Blum&Mitchell, 1998) can improve the performance of prompt-based learning by using unlabeled data. While prompting has emerged as a promising paradigm for few-shot and zero-shot learning, it is often brittle and requires much larger models compared to the standard supervised setup. We find that co-training makes it possible to improve the original prompt model and at the same time learn a smaller, downstream task-specific model. In the case where we only have partial access to a prompt model (e.g., output probabilities from GPT-3 (Brown et al., 2020)) we learn a calibration model over the prompt outputs. When we have full access to the prompt model's gradients but full finetuning remains prohibitively expensive (e.g., T0 (Sanh et al., 2021)), we learn a set of soft prompt continuous vectors to iteratively update the prompt model. We find that models trained in this manner can significantly improve performance on challenging datasets where there is currently a large gap between prompt-based learning and fully-supervised models.

Subfields: Artificial Intelligence, Large Language Models, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Prompting & Reasoning

Citations in the corpus

Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (2022, methodology): “Prompt tuning has been applied to various applications for NLP including continual learning [59], model robustness [60, 61], summarization [62], machine translation [63], co-training [64], probing language models [65, 65], inverse prompting [66], and transfer learning [67].”
TabLLM: Few-shot Classification of Tabular Data with Large Language Models (2022, unknown): “Prior work has shown that medical-domain-speciﬁc language models, such as PubMedBERT, and general-domain models with medical data in their training sets, such as GPT-3, perform well at downstream prediction tasks on medical data even with fairly few samples (Gu et al., 2021; Agrawal et al., 2022).”
Large language models are few-shot clinical information extractors (2022, methodology): “To evaluate on MIMIC we distill GPT-3 + R into a smaller model by treating the outputs as weak supervision and following Lang et al. (2022b) “+ distillation”, then evaluate the smaller model on MIMIC as well. being transferred to a new domain, our weakly-supervised PubMedBERT model performs…”
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model (2022, background): “Of particular interest are approaches for ﬁltering language model outputs that are unlikely to be correct, which could result in a cleaner silver training set (Lang et al., 2022; Smith et al., 2022).”