Memory-assisted prompt editing to improve GPT-3 after deployment
By Aman Madaan, Niket Tandon, Peter Clark, Yiming Yang. 2022
194 citations recorded by the source corpus.
Large LMs such as GPT -3 are powerful, but can commit mistakes that are obvious to humans. For example, GPT -3 would mistakenly interpret "What word is similar to good ?" to mean a homonym, while the user intended a synonym. Our goal is to effectively correct such errors via user interactions with the sys-tem but without retraining, which will be pro-hibitively costly. We pair GPT -3 with a growing memory of recorded cases where the model misunderstood the user’s intents, along with user feedback for clarification. Such a memory allows our system to produce enhanced prompts for any new query based on the user feedback for error correction on similar cases in the past. On four tasks (two lexical tasks, two advanced ethical reasoning tasks), we show how a (simulated) user can interactively teach a deployed GPT -3, substantially increasing its accuracy over the queries with different kinds of misunderstandings by the GPT -3. Our approach is a step towards the low-cost utility enhancement for very large pre-trained LMs. 1
Subfields: Artificial Intelligence, Large Language Models, Alignment & RLHF
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Graham Neubig
- Mohit Bansal
- Luke S. Zettlemoyer
- Hannaneh Hajishirzi
- Percy Liang
- Yejin Choi
- Noah A. Smith
In Large Language Models
- Jianfeng Gao
- Weizhu Chen
- Pengcheng He
- Graham Neubig
- Noah A. Smith
- Hannaneh Hajishirzi
- Michel Galley
- Dilek Z. Hakkani-Tür
In Alignment & RLHF
- Jianfeng Gao
- Mari Ostendorf
- Noah A. Smith
- Prithviraj Ammanabrolu
- Yushi Hu
- Hannaneh Hajishirzi
- Weijia Shi
- Alane Suhr
Citations in the corpus
- Editing Large Language Models: Problems, Methods, and Opportunities (2023, unknown): “MemPrompt (Madaan et al., 2022),IKE (Zheng et al., 2023) and MeLLo (Zhong et al., 2023).”
- OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement (2024, unknown): “For many sequence generation tasks, iterative approaches are often taken to improve the generation quality, e.g., script generation (Tandon et al., 2021), summarization (Scheurer et al., 2022), and other tasks as shown in (Madaan et al., 2022; Saunders et al., 2022).”
- Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback (2023, background): “More recent work attempts to combine black-box LLMs with external knowledge, such as incorporating external knowledge into prompts (Madaan et al., 2022; Lazaridou et al., 2022), making GPT-3 more faithful (He et al., 2022), and combining web knowledge with GPT-3 (Nakano et al., 2021).”
- Rethinking machine unlearning for large language models (2024, unknown): “Thus, existing memory-based model editing techniques (Li et al., 2022; Mitchell et al., 2022; Madaan et al., 2022; Zheng et al., 2023) focus on updating short-term working memory instead of altering the long-term memory encapsulated in the model’s weights.”
- A survey on large language model based autonomous agents (2023, background): “In MemPrompt [106], the users are requested to provide feedback in natural language regarding the problem-solving intentions of the agent, and this feedback is stored in memory.”
- DialGuide: Aligning Dialogue Model Behavior with Developer Guidelines (2022, background): “…work has explored editing models by computing targeted changes in the model’s parameters (Sinitsin et al., 2020; Hase et al., 2021; Mitchell et al., 2021; Meng et al., 2022), while others have explored natural language feedback (Madaan et al., 2022; Scheurer et al., 2022; Zeidler et al., 2022).”
- Fine-Grained Human Feedback Gives Better Rewards for Language Model Training (2023, methodology): “[23, 6] collect and store NL human feedback in a feedback memory for the model to retrieve and then perform the end task conditioning on the retrieved feedback.”
- A Brief Overview of ChatGPT: The History, Status Quo and Potential Future Development (2023, methodology): “Thus, RLHF has been positively applied to fine-tuning language models in most domains in natural language process (NLP) such as dialogue [88], [89], translation [90], [91], story generation [92], evidence extraction [93], [94], and semantic parsing [95].”
- Learning Performance-Improving Code Edits (2023, methodology): “Given the high costs of training models through the OpenAI API, we also use this dataset as a base for fine-tuning experiments with GPT -3.5.”
- When large language models meet personalization: perspectives of challenges and opportunities (2023, unknown): “MemPrompt [231] enhances the prompt by incorporating a memory module, enabling GPT-3 to possess stronger long-dialogue memory capability.”
- A Survey of Large Language Models (2023, unknown): “By incorporating the extracted relevant information into the context [360, 361], LLMs can acquire new factual knowledge and perform better on relevant tasks.”