On Robust Prefix-Tuning for Text Classification

By Zonghan Yang, Yang Liu. 2022

23 citations recorded by the source corpus.

Recently, prefix-tuning has gained increasing attention as a parameter-efficient finetuning method for large-scale pretrained language models. The method keeps the pretrained models fixed and only updates the prefix token parameters for each downstream task. Despite being lightweight and modular, prefix-tuning still lacks robustness to textual adversarial attacks. However, most currently developed defense techniques necessitate auxiliary model update and storage, which inevitably hamper the modularity and low storage of prefix-tuning. In this work, we propose a robust prefix-tuning framework that preserves the efficiency and modularity of prefix-tuning. The core idea of our framework is leveraging the layerwise activations of the language model by correctly-classified training data as the standard for additional prefix finetuning. During the test phase, an extra batch-level prefix is tuned for each batch and added to the original prefix for robustness enhancement. Extensive experiments on three text classification benchmarks show that our framework substantially improves robustness over several strong baselines against five textual attacks of different types while maintaining comparable accuracy on clean texts. We also interpret our robust prefix-tuning framework from the optimal control perspective and pose several directions for future research.

Subfields: Artificial Intelligence, Natural Language Processing, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhiyuan Liu
2. Maosong Sun
3. Mohit Bansal
4. Yang Liu
5. Jianfeng Gao
6. Hannaneh Hajishirzi
7. Graham Neubig
8. Luke S. Zettlemoyer

In Natural Language Processing

1. Zhiyuan Liu
2. Maosong Sun
3. Mohit Bansal
4. Yang Liu
5. Colin Raffel
6. Graham Neubig
7. Caiming Xiong
8. Haokun Liu

In Efficiency & Compression

1. Mohit Bansal
2. Haitao Zheng
3. Zhiding Yu
4. Maosong Sun
5. Zonghan Yang
6. Tom Goldstein
7. Anima Anandkumar
8. Weilin Zhao

Citations in the corpus

• Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (2022, methodology): “Prompt tuning has been applied to various applications for NLP including continual learning [59], model robustness [60, 61], summarization [62], machine translation [63], co-training [64], probing language models [65, 65], inverse prompting [66] and transfer learning [67].”
• Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models (2022, background): “Prompt tuning [12, 25–27] proposes to learn the prompt from downstream data in the continual input embedding space, which presents a parameter-efficient way of fine-tuning foundation models.”
• Parameter-efficient fine-tuning of large-scale pre-trained language models (2023, background): “For example, robust prefix-tuning 48 tunes additional layer-wise prefix parameters during inference.”