When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels

By Weiyan Shi, Emily Dinan, Kurt Shuster, Jason Aaron Edward Weston, Jing Xu. 2022

23 citations recorded by the source corpus.

Deployed dialogue agents have the potential to integrate human feedback to continuously improve themselves. However, humans may not always provide explicit signals when the chatbot makes mistakes during interactions. In this work, we propose Juicer, a framework to make use of both binary and free-form textual human feedback. It works by: (i) extending sparse binary feedback by training a satisfaction classifier to label the unlabeled data; and (ii) training a reply corrector to map the bad replies to good ones. We find that augmenting training with model-corrected replies improves the final dialogue model, and we can further improve performance by using both positive and negative replies through the recently proposed Director model.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Constitutional AI: Harmlessness from AI Feedback (2022, background): “Some other recent works on self-supervision include [Shi et al., 2022, Huang et al., 2022].”
Fine-Grained Human Feedback Gives Better Rewards for Language Model Training (2023, background): “[38, 14, 42] train a conversational model to predict both the response and a binary user satisfaction score in order to improve the response generation.”
AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback (2023, methodology): “However, there exist methods in the literature other than those explored in AlpacaFarm that can incorporate alternative sources of feedback such as natural language [74, 32, 22, 59, 55, 13, 56, 40], numeric ratings [44, 31], or execution traces [13].”
Training Language Models with Language Feedback at Scale (2023, methodology): “Liu et al. (2022); Schick et al. (2022) train LMs to refine outputs based on feedback (without finetuning on the refinements), an approach that improves results when incorporated into ILF, as shown in subsequent work to ours (Shi et al., 2022).”