Learning from Dialogue after Deployment: Feed Yourself, Chatbot!

By Braden Hancock, Antoine Bordes, Pierre-Emmanuel Mazaré, Jason Aaron Edward Weston. 2019

221 citations recorded by the source corpus.

The majority of conversations a dialogue agent sees over its lifetime occur after it has already been trained and deployed, leaving a vast store of potential training signal untapped. In this work, we propose the self-feeding chatbot, a dialogue agent with the ability to extract new training examples from the conversations it participates in. As our agent engages in conversation, it also estimates user satisfaction in its responses. When the conversation appears to be going well, the user’s responses become new training examples to imitate. When the agent believes it has made a mistake, it asks for feedback; learning to predict the feedback that will be given improves the chatbot’s dialogue abilities further. On the PersonaChat chit-chat dataset with over 131k training examples, we find that learning from dialogue with a self-feeding chatbot significantly improves performance, regardless of the amount of traditional supervision.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, background): “Dialog based learning (Weston, 2016; Li et al., 2016; Hancock et al., 2019) develops techniques to learn from interactive natural language feedback in dialog.”
Training language models to follow instructions with human feedback (2022, methodology): “This work is in turn influenced by similar work using human feedback as a reward in domains such as dialogue (Jaques et al., 2019; Yi et al., 2019; Hancock et al., 2019), translation (Kreutzer et al.”
Fine-Tuning Language Models from Human Preferences (2019, background): “RL is not the only way to incorporate ongoing human feedback: Hancock et al. (2019) ask humans what a dialogue system should have said instead, then continue supervised training.”
Learning to summarize from human feedback (2020, methodology): “We treat the logit output of the reward model as a reward that we optimize using reinforcement learning, speciﬁcally with the PPO algorithm [58].”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “This has been explored in Hancock et al. (2019), where a “selffeeding chatbot” learns to estimate its partner’s satisfaction and can also ask for feedback when it believes it has made a mistake, see Figure 1.”
Recursively Summarizing Books with Human Feedback (2021, background): “, 2020), dialogue (Jaques et al., 2019; Yi et al., 2019; Hancock et al., 2019), translation (Kreutzer et al.”
When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels (2022, methodology): “Prior work has studied how to collect and learn from feedback in human-model dialogues (Li et al., 2016a,b; Hancock et al., 2019; Xu et al., 2022).”
Training Language Models with Language Feedback (2022, background): “A few prior works also learn from language feedback, for the purpose of ranking candidate outputs rather than generating outputs (Weston, 2016; Li et al., 2016; Hancock et al., 2019; Li et al., 2022).”
Human-centric Dialog Training via Offline Reinforcement Learning (2020, neutral): “Although implicit signals such as sentiment (Hancock et al., 2019) and conversation length (Zhou et al., 2018) have been used in maximum likelihood estimation (MLE) systems, the idea of using such signals as a reward for RL is relatively unexplored.”
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage (2022, methodology): “…with appropriate learning systems, could then potentially keep improving over time (Carlson et al., 2010; Kiela et al., 2021; Agichtein et al., 2006; Liu et al., 2021; Madotto et al., 2021; Shuster et al., 2021b), where (Hancock et al., 2019) refer to this approach as a self-feeding chatbot.”
Survey of Hallucination in Natural Language Generation (2022, background): “The PersonaChat datasets [31, 228] have accelerated research into persona consistency [68, 91, 126, 208, 219, 224, 232].”
Recipes for Building an Open-Domain Chatbot (2020, background): “We note that these models cannot be taught a concept through further conversation, so as-is they will always be stunted, see (Weston, 2016; Hancock et al., 2019) for early work in this direction.”
Dynabench: Rethinking Benchmarking in NLP (2021, background): “A natural setting for exploring these ideas might be dialogue (Hancock et al., 2019; Shuster et al., 2020).”
Lifelong and Continual Learning Dialogue Systems: Learning during Conversation (2021, background): “Some initial attempts have been made in (Mazumder et al. 2020a,b, 2019; Hancock et al. 2019; Luo et al. 2019).”
SaFeRDialogues: Taking Feedback Gracefully after Conversational Safety Failures (2021, background): “This could in turn be precious training signal for on-going improvement of models through online learning (Hancock et al., 2019; Roller et al., 2020a).”
Deploying Lifelong Open-Domain Dialogue Learning (2020, methodology): “Closer to our work, is the self-feeding chatbot (Hancock et al., 2019), whereby it is shown that models can be used to collect data to improve themselves via crowdsourcing utilizing the PersonaChat task.”