Deploying Lifelong Open-Domain Dialogue Learning

By Kurt Shuster, Jack Urbanek, Emily Dinan, Arthur Szlam, Jason Aaron Edward Weston. 2020

24 citations recorded by the source corpus.

Much of NLP research has focused on crowdsourced static datasets and the supervised learning paradigm of training once and then evaluating test performance. As argued in de Vries et al. (2020), crowdsourced data has the issues of lack of naturalness and relevance to real-world use cases, while the static dataset paradigm does not allow for a model to learn from its experiences of using language (Silver et al., 2013). In contrast, one might hope for machine learning systems that become more useful as they interact with people. In this work, we build and deploy a role-playing game, whereby human players converse with learning agents situated in an open-domain fantasy world. We show that by training models on the conversations they have with humans in the game the models progressively improve, as measured by automatic metrics and online engagement scores. This learning is shown to be more efficient than crowdsourced data when applied to conversations with real users, as well as being far cheaper to collect.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels (2022, background): “Past research has also explored how to integrate feedback into dialogue agents (Li et al., 2016a,b; Hancock et al., 2019; Shuster et al., 2020; Xu et al., 2022).”
Continual Learning in Task-Oriented Dialogue Systems (2020, background): “Finally, (Shuster et al., 2020) showed that by training models on humans-machine conversations in a open-domain fantasy world game (Fan et al., 2020) the models progressively improve, as measured by automatic metrics and online engagement scores.”
Reason first, then respond: Modular Generation for Knowledge-infused Dialogue (2021, background): “(Shuster et al., 2020) as well as a derived version of it, LightQA, ending on a question about the episode.”
Understanding and predicting user dissatisfaction in a neural generative chatbot (2021, methodology): “In symmetric settings such as the role-playing game LIGHT (Shuster et al., 2020), the user utterances themselves can be used to retrain the dialogue agent.”
Learning from data in the mixed adversarial non-adversarial case: Finding the helpers and ignoring the trolls (2022, background): “The promise of this approach is that the distribution of data will more closely match those organic users’ desires, rather than decided by the researchers themselves when creating datasets (Gabriel et al., 2020; Roller et al., 2020; Shuster et al., 2020; Ouyang et al., 2022).”
Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents (2022, background): “Evaluation by deployment can be successful (Gabriel et al., 2020; Shuster et al., 2020), where behavioral metrics such as the amount of conversation time per user or retention rate can serve as a proxy for interestingness and engagingness metrics.”
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback (2022, background): “Deployed models can be improved in symmetric conversations conducted between models and humans during deployment by learning to mimic human conversationalists, as shown in the LIGHT dialogue game (Shuster et al., 2020).”