Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback

By Jing Xu, Megan Ung, Mojtaba Komeili, Kushal Arora, Y-Lan Boureau, Jason Aaron Edward Weston. 2022

47 citations recorded by the source corpus.

Frozen models trained to mimic static datasets can never improve their performance. Models that can employ internet-retrieval for up-to-date information and obtain feedback from humans during deployment provide the promise of both adapting to new information, and improving their performance. In this work we study how to improve internet-driven conversational skills in such a learning framework. We collect deployment data, which we make publicly available, of human interactions, and collect various types of human feedback – including binary quality measurements, free-form text feedback, and fine-grained reasons for failure. We then study various algorithms for improving from such feedback, including standard supervised learning, rejection sampling, model-guiding and reward-based learning, in order to make recommendations on which type of feed- back and algorithms work best. We find the recently introduced DIRECTOR model (Arora et al., 2022) shows significant improvements over other existing approaches.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

Citations in the corpus

Augmented Language Models: a Survey (2023, background): “Human feedback can be exploited to improve the quality of machine-generated text, for example for dialog agents (Xu et al., 2022).”
When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels (2022, methodology): “A.5.1 Oracle performance using 100% feedback data Xu et al. (2022) trained various methods on the entire FITS dataset.”
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage (2022, methodology): “This will require new research, and while we have made steps in this direction (Ju et al., 2022; Xu et al., 2022b), much work remains.”
The rise and potential of large language model based agents: a survey (2025, unknown): “Xu et al. [361] compared various types of feedback and observed that combining multiple types of feedback can yield better results.”
Fine-Grained Human Feedback Gives Better Rewards for Language Model Training (2023, background): “[38, 14, 42] train a conversational model to predict both the response and a binary user satisfaction score in order to improve the response generation.”
Training Language Models with Language Feedback at Scale (2023, background): “A few prior works also learn from language feedback for the purpose of ranking candidate outputs rather than generating outputs (Weston, 2016; Li et al., 2016; Hancock et al., 2019; Li et al., 2022; Xu et al., 2022).”