SwitchPrompt: Learning Domain-Specific Gated Soft Prompts for Classification in Low-Resource Domains

By Koustava Goswami, Lukas Lange, Jun Araki, Heike Adel. 2023

19 citations recorded by the source corpus.

Prompting pre-trained language models leads to promising results across natural language processing tasks but is less effective when applied in low-resource domains, due to the domain gap between the pre-training data and the downstream task. In this work, we bridge this gap with a novel and lightweight prompting methodology called SwitchPrompt for the adaptation of language models trained on datasets from the general domain to diverse low-resource domains. Using domain-specific keywords with a trainable gated prompt, SwitchPrompt offers domain-oriented prompting, that is, effective guidance on the target domains for general-domain language models. Our few-shot experiments on three text classification benchmarks demonstrate the efficacy of the general-domain pre-trained language models when used with SwitchPrompt. They often even outperform their domain-specific counterparts trained with baseline state-of-the-art prompting methods by up to 10.7% performance increase in accuracy. This result indicates that SwitchPrompt effectively reduces the need for domain-specific language model pre-training.

Subfields: Prompting & Reasoning, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Prompting & Reasoning

1. Yu Qiao
2. Peng Gao
3. Renrui Zhang
4. Shilin Yan
5. Pan Lu
6. Hongsheng Li

In Efficiency & Compression

1. Yu Qiao
2. Peng Gao
3. Hongsheng Li
4. Aojun Zhou

Citations in the corpus

• LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, methodology): “Moreover, we verify the effectiveness of our approach to fine-tune large models in other domains.”