Dialogue Learning With Human-In-The-Loop

By Jiwei Li, Alexander H. Miller, Sumit Chopra, Marc'Aurelio Ranzato, Jason Aaron Edward Weston. 2016

142 citations recorded by the source corpus.

An important aspect of developing conversational agents is to give a bot the ability to improve through communicating with humans and to learn from the mistakes that it makes. Most research has focused on learning from fixed training sets of labeled data rather than interacting with a dialogue partner in an online fashion. In this paper we explore this direction in a reinforcement learning setting where the bot improves its question-answering ability from feedback a teacher gives following its generated responses. We build a simulator that tests various aspects of such learning in a synthetic environment, and introduce models that work in this regime. Finally, real experiments with Mechanical Turk validate the approach.

Subfields: Artificial Intelligence, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Fine-Tuning Language Models from Human Preferences (2019, neutral): “Li et al. (2016) reached similar conclusions in a restricted dialogue setting after validating in simulation that online and batched trained performed similarly.”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, methodology): “Similarly, generative models typically report perplexity of a held-out test set (e.g. Li et al. (2017c); Dinan et al. (2020); Shuster et al. (2020); Fan et al. (2019); Adiwardana et al. (2020); Roller et al. (2020)), and recent work has even found perplexity correlates strongly with human…”
Learning from Dialogue after Deployment: Feed Yourself, Chatbot! (2019, background): “Li et al. (2017b) considered this question as well, but again in the context of a QA setting rather than dialogue.”
When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels (2022, background): “Prior work has studied how to collect and learn from feedback in human-model dialogues (Li et al., 2016a,b; Hancock et al., 2019; Xu et al., 2022).”
Training Language Models with Language Feedback (2022, background): “A few prior works also learn from language feedback, for the purpose of ranking candidate outputs rather than generating outputs (Weston, 2016; Li et al., 2016; Hancock et al., 2019; Li et al., 2022).”
Human-centric Dialog Training via Offline Reinforcement Learning (2020, background): “RL in the open-domain dialog setting is less explored (Li et al., 2016, 2017b, 2018).”
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage (2022, background): “Li et al. (2016b) studies models that learn how to sometimes ask appropriate questions in order to learn from the answers, while Li et al. (2016a) learns from general textual feedback/comments from the user, particularly in the case where the bot has produced a low quality response.”
Neural Approaches to Conversational AI (2018, neutral): “erstanding and Dialogue State Tracking NLU and DST are two closely related components essential to a dialogue system. They can have a signiﬁcant impact on the overall system’s performance (see, e.g., Li et al. (2017e)). This section reviews some of the classic and state-of-the-art approaches. 4.3.1 ”
Bootstrapping a Neural Conversational Agent with Dialogue Self-Play, Crowdsourcing and On-Line Reinforcement Learning (2018, background): “Recently proposed neural conversational models (Vinyals and Le (2015)) are trained with supervision over a large corpus of dialogues (Ser-ban et al. (2016, 2017); Lowe et al. (2017)) or with reinforcement to optimize a long term reward (Li et al. (2016a,b)).”
WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, neutral): “It is also common to split the data to induce a shift that is more extreme than one might typically encounter, e.g., in PACS (Li et al., 2017), which includes classifying photos solely from cartoons and other stylized representations; DeepFashion Remixed (Hendrycks et al., 2020), which involves…”
Deploying Lifelong Open-Domain Dialogue Learning (2020, methodology): “A number of crowdsourced or scraped datasets have been developed to that end, including Daily Dialogue (Li et al., 2017), Per-sonaChat (Li et al., 2016a), Empathetic Dialogues (Rashkin et al., 2019) and Wizard of Wikipedia (Dinan et al., 2019c).”
A Survey on Dialogue Systems: Recent Advances and New Frontiers (2017, background): “[33] further attempted to improve the bot’s ability to learn from interaction.”
Latent Intention Dialogue Models (2017, background): “As in reinforcement learning, which has been a very common learning paradigm in dialogue systems (Gaˇsi´c et al., 2013; Su et al., 2016; Li et al., 2017), the policy is also parameterised by a discrete set of actions.”
Deep Reinforcement Learning: An Overview (2017, background): “Li et al. (2017) proposed to improve sequence generation by considering the knowledge about the future.”
Lifelong and Interactive Learning of Factual Knowledge in Dialogues (2019, background): “on rules and information retrieval techniques (Banchs and Li, 2012; Ameixa et al., 2014; Lowe et al., 2015; Serban et al., 2015). Recent deep learning models (Vinyals and Le, 2015; Xing et al., 2017; Li et al., 2017c) learn from large corpora. However, since they do not use explicit knowledge bases ”
AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback (2023, methodology): “However, there exist methods in the literature other than those explored in AlpacaFarm that can incorporate alternative sources of feedback such as natural language [72, 32, 22, 57, 53, 13, 54, 39], numeric ratings [43, 31], or execution traces [13].”
Dialogue in the Wild: Learning from a Deployed Role-Playing Game with Humans and Bots (2021, methodology): “Related approaches have also been applied to the cases of question answering (Li et al., 2016a,b), and in simulators (Mazumder et al., 2019; Nguyen and Daumé III, 2019) as well.”