RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning

By Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Han Guo, Tianmin Shu, Meng Song, Eric P. Xing, Zhiting Hu. Conference on Empirical Methods in Natural Language Processing, 2022

520 citations recorded by the source corpus.

Prompting has shown impressive success in enabling large pre-trained language models (LMs) to perform diverse NLP tasks, especially with only few downstream data. Automatically finding the optimal prompt for each task, however, is challenging. Most existing work resorts to tuning *soft* prompts (e.g., embeddings) which fall short of interpretability, reusability across LMs, and applicability when gradients are not accessible. *Discrete* prompts, on the other hand, are difficult to optimize, and are often created by “enumeration (e.g., paraphrasing)-then-selection” heuristics that do not explore the prompt space systematically. This paper proposes RLPrompt, an efficient discrete prompt optimization approach with reinforcement learning (RL). RLPrompt formulates a parameter-efficient policy network that generates the optimized discrete prompt after training with reward. To harness the complex and stochastic reward signals from the large LM environment, we incorporate effective reward stabilization that substantially enhances training efficiency. RLPrompt is flexibly applicable to different types of LMs, such as masked (e.g., BERT) and left-to-right models (e.g., GPTs), for both classification and generation tasks. Experiments on few-shot classification and unsupervised text style transfer show superior performance over a wide range of existing fine-tuning or prompting methods. Interestingly, the resulting optimized prompts are often ungrammatical gibberish text; and surprisingly, those gibberish prompts are transferrable between different LMs to retain significant performance, indicating that LM prompting may not follow human language patterns.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/07759a84f27e43cfa5bc8d579f8227c96e6ae1dc

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Transfer Visual Prompt Generator across LLMs (2023, background): “Another way of soft prompt transfer [9] is to conduct prompt tuning on discrete prompts, and thus the discrete prompts can be directly shared across models.”
Fine-Tuning Language Models with Just Forward Passes (2023, background): “[24] Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric Xing, and Zhiting Hu.”
Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery (2023, methodology): “…optimization frameworks for transformer language models and subsequent approaches have included a gradient-free phrase editing method (Prasad et al., 2022), an embedding optimization approach based on Langevin dynamics (Shi et al., 2022) and a reinforcement learning approach (Deng et al., 2022).”
Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering (2022, background): “…et al., 2021), summarization (Stiennon et al., 2020; Paulus et al., 2018), machine translation (Shen et al., 2016; Wu et al., 2016), grounded text generation (Ammanabrolu et al., 2021, 2022), controlled text generation (Lu et al., 2022), and prompt generation (Guo et al., 2021; Deng et al., 2022).”
Demystifying Prompts in Language Models via Perplexity Estimation (2022, background): “Other work also focuses on discrete prompts, aiming to improve zero-shot performance (Gao et al., 2021; Le Scao and Rush, 2021; Deng et al., 2022), or trains continuous prompts (Li and Liang, 2021; Lester et al.”
Large Language Models as Optimizers (2023, background): “…(Lester et al., 2021; Li & Liang, 2021; Liu et al., 2021; Qin & Eisner, 2021), as well as performing discrete prompt optimization by gradient-guided search (Shin et al., 2020; Wen et al., 2023; Gao et al., 2020; Chen et al., 2023d) and reinforcement learning (Deng et al., 2022; Zhang et al., 2023).”
Unleashing the potential of prompt engineering in Large Language Models: a comprehensive review (2023, background): “For another example, many studies experiment mainly on certain models or tasks and employ disparate evaluation metrics, restricting comparability across meth-ods [110, 111].”
Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting (2023, methodology): “, 2023), reinforcement learning (Deng et al., 2022), and by using LLMs themselves (Zhou et al.”
Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers (2024, unknown): “In spite of their effective performance, two drawbacks of such paradigms can not be ignored: 1) The optimization of continuous prompts requires parameters of LLMs that are inaccessible for black-box APIs.”