PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts

By Stephen H. Bach, Victor Sanh, Zheng-Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Févry and 16 more. 2022

417 citations recorded by the source corpus.

PromptSource is a system for creating, sharing, and using natural language prompts. Prompts are functions that map an example from a dataset to a natural language input and target output. Using prompts to train and query language models is an emerging area in NLP that requires new tools that let users develop and refine these prompts collaboratively. PromptSource addresses the emergent challenges in this new setting with (1) a templating language for defining data-linked prompts, (2) an interface that lets users quickly iterate on prompt development by observing outputs of their prompts on many examples, and (3) a community-driven set of guidelines for contributing new prompts to a common pool. Over 2,000 prompts for roughly 170 datasets are already available in PromptSource. PromptSource is available at https://github.com/bigscience-workshop/promptsource.

Subfields: Artificial Intelligence, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Prompting & Reasoning

Citations in the corpus

Efficient Large Scale Language Modeling with Mixtures of Experts (2021, background): “ing) before careful prompting can improve task performance, that grows with model scale without any fine-tuning, and this gave rise to new resources for prompt engineering (Bach et al., 2022).”
OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization (2022, background): “…from multiple existing work on instruction-tuning : FLAN (Wei et al., 2022a), T0 (Sanh et al., 2022); prompt crowdsourcing : PromptSource (Bach et al., 2022); cross-task transfer studies: ExMix (Aribandi et al., 2022), T5 (Raffel et al., 2020), CrossFit (Ye et al., 2021); and…”
Self-Instruct: Aligning Language Models with Self-Generated Instructions (2022, background): “Additionally, to compare SELF-INSTRUCT training with other publicly available instruction tuning data, we further finetune GPT3 model with data from PROMPTSOURCE and SUPERNI, which are used to train the T0 and T𝑘-INSTRUCT models.”
Rationale-Augmented Ensembles in Language Models (2022, background): “, 2021), how each task is verbalized (Bach et al., 2022), and the distribution of labels in the prompts (Zhao et al.”
QLoRA: Efficient Finetuning of Quantized LLMs (2023, methodology): “Approaches and datasets include MetaICL [40], MetaTuning [74], InstructGPT [43], FLAN [63, 12], PromptSource [3], Super-NaturalInstructions [62, 51], Self-instruct [60], UnnaturalInstructions [26], OPT-IML [29], UnifiedSKG[68], OIG/Chip2 [32], Alpaca [56], Vicuna [10], Koala [20], and…”
LongForm: Effective Instruction Tuning with Reverse Instructions (2023, positive): “We also employ the BEA-2019 for grammatical error correction (Bryant et al., 2019); incorrect inputs and template instruction are the input, the correct output the target.”
GLM-130B: An Open Bilingual Pre-trained Model (2022, methodology): “Datasets from T0PromptSource (Sanh et al., 2022; Bach et al., 2022) are named in their Hugging Face datasets identifiers.”
Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (2022, methodology): “Examples in the datasets used to train T0 were prompted by applying the prompt templates from the Public Pool of Prompts (P3 [35]), which convert each example in each dataset to a prompted text-to-text format where each label corresponds to a different string.”
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, background): “The subfield of language models learning instruction-following capabilities aims to generate responses based on natural language commands, which have been extensively researched in language [64, 63, 3, 46], and multi-modality [59, 42] domains.”
The rise and potential of large language model based agents: a survey (2025, positive): “With just task descriptions or demonstrations, they can eﬀectively handle previously unseen tasks [18, 87, 478].”
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey (2021, background): “T0 [159] also fine-tunes on instruction datasets, with a particular focus on evaluating zero-shot performance on held-out tasks and robustness to prompt wording, and comes with a large collection of prompts for diverse datasets curated via PromptSource [6].”
Instruction Tuning for Large Language Models: A Survey (2023, neutral): “…of a dataset should be guided by the target application: human-crafted datasets may be preferable for safety-critical tasks due to their quality control over instruction clarity and response correctness, whereas distillation-based datasets offer the breadth needed for general-purpose assistants.”
Unsupervised Cross-Task Generalization via Retrieval Augmentation (2022, methodology): “We observe that sometimes even though T0-like models (including ours) answer the input questions correctly, their raw outputs are not exactly the same as the truth outputs generated by the PromptSource templates.”
Fine-Tuning Language Models with Just Forward Passes (2023, methodology): “[5] Stephen H Bach, Victor Sanh, Zheng-Xin Yong, Albert Webson, Colin Raffel, Nihal V Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, et al.”