Learning to repair: Repairing model output errors after deployment using a dynamic memory of feedback

By Niket Tandon, Aman Madaan, Peter Clark, Yiming Yang. 2021

45 citations recorded by the source corpus.

Large language models (LMs), while powerful, are not immune to mistakes, but can be difficult to retrain. Our goal is for an LM to continue to improve after deployment, without retraining, using feedback from the user. Our approach pairs an LM with (i) a growing memory of cases where the user identified an output error and provided general feedback on how to correct it (ii) a corrector model, trained to translate this general feedback into specific edits to repair the model output. Given a new, unseen input, our model can then use feedback from similar, past cases to repair output errors that may occur. We instantiate our approach using an existing, fixed model for script generation, that takes a goal (e.g.,"bake a cake") and generates a partially ordered sequence of actions to achieve that goal, sometimes containing errors. Our memory-enhanced system, FBNet, learns to apply user feedback to repair such errors (up to 30 points improvement), while making a start at avoiding similar past mistakes on new, unseen examples (up to 7 points improvement in a controlled setting). This is a first step towards strengthening deployed models, potentially broadening their utility. Our code and data is available at https://github.com/allenai/interscript/.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

Citations in the corpus

Self-Refine: Iterative Refinement with Self-Feedback (2023, background): “Humans have been an effective source of feedback (Tandon et al., 2021; Elgohary et al., 2021; Tandon et al., 2022; Bai et al., 2022a).”
When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels (2022, methodology): “Tandon et al. (2022) trained a memory-augmented corrector to convert feedback to edits and ﬁx model outputs for a script generation task.”
The rise and potential of large language model based agents: a survey (2025, unknown): “Additionally, agents can utilize a memory module to store feedback for future reuse [359].”