ZeroPrompt: Scaling Prompt-Based Pretraining to 1, 000 Tasks Improves Zero-Shot Generalization

By Hanwei Xu, Yujun Chen, Yulun Du, Nan Shao, Haiyu Li, Zhilin Yang. 2022

76 citations recorded by the source corpus.

We propose a multitask pretraining approach ZeroPrompt for zero-shot generalization, focusing on task scaling and zero-shot prompting. While previous models are trained on only a few dozen tasks, we scale to 1,000 tasks for the first time using real-world data. This leads to a crucial discovery that task scaling can be an efficient alternative to model scaling; i.e., the model size has little impact on performance with an extremely large number of tasks. Our results show that task scaling can substantially improve training efficiency by 30 times in FLOPs. Moreover, we present a prompting method that incorporates a genetic algorithm to automatically search for the best prompt for unseen tasks, along with a few other improvements. Empirically, ZeroPrompt substantially improves both the efficiency and the performance of zero-shot learning across a variety of academic and production datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Prompting & Reasoning

Citations in the corpus

Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “[131] Hanwei Xu, Yujun Chen, Yulun Du, Nan Shao, Yanggang Wang, Haiyu Li, and Zhilin Yang.”
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks (2022, critical): “This ﬁnding contradicts the claim in Xu et al. (2022) that “model size has little impact on performance with an extremely large number of tasks.””
What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? (2022, background): “One recent line of work [Sanh et al., 2021, Wei et al., 2021, Xu et al., 2022] has demonstrated that adding an explicit multitask ﬁnetuning stage on an ensemble of prompted tasks after pretraining can signiﬁcantly boost the zero-shot capabilities of LLMs. Modern LLMs are based on the Transformer…”
InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning (2022, background): “Going beyond multi-task training, these approaches show better generalization to unseen tasks when prompted with a few examples (Bragg et al., 2021; Min et al., 2022a,b) or language definitions and constraints (Weller et al., 2020; Zhong et al., 2021b; Xu et al., 2022).”
Benchmarking Generalization via In-Context Instructions on 1, 600+ Language Tasks (2022, background): “, 2021) or by language definitions (Efrat and Levy, 2020; Weller et al., 2020; Zhong et al., 2021; Mishra et al., 2022b,a; Xu et al., 2022).”
WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct (2023, unknown): “Subsequently, instruction fine-tuning work such as FLAN [87], ExT5 [88], T0 [89], UnifiedQA [90], ZeroPrompt [91], and FLAN-T5 [92] emerged to adapt for a large number of downstream tasks.”
Harnessing the Power of David against Goliath: Exploring Instruction Data Generation without Using Closed-Source Models (2023, methodology): “, ZeroPrompt (Xu et al., 2022), Super-Natural Instructions (Wang et al.”
Improving In-Context Few-Shot Learning via Self-Supervised Training (2022, methodology): “…for in-context fewshot or zero-shot learning using human-annotated datasets (Ye et al., 2021; Wei et al., 2022; Sanh et al., 2022; Min et al., 2021; Xu et al., 2022) via instructions (Weller et al., 2020; Efrat and Levy, 2020; Mishra et al., 2021; Ouyang et al., 2022) and retrieved examples (Hu…”
Dataset Debt in Biomedical Language Modeling (2022, background): “(Zhao et al., 2022) curated sci-080 entific corpora to identify patient-specific infor-081 mation (e.g., mining PubMed Central to identify case reports that respect licensing for re-use and re-083 distribution).”