Dialog-based Language Learning

By Jason Aaron Edward Weston. 2016

112 citations recorded by the source corpus.

A long-term goal of machine learning research is to build an intelligent dialog agent. Most research in natural language understanding has focused on learning from fixed training sets of labeled data, with supervision either at the word level (tagging, parsing tasks) or sentence level (question answering, machine translation). This kind of supervision is not realistic of how humans learn, where language is both learned by, and used for, communication. In this work, we study dialog-based language learning, where supervision is given naturally and implicitly in the response of the dialog partner during the conversation. We study this setup in two domains: the bAbI dataset of (Weston et al., 2015) and large-scale question answering from (Dodge et al., 2015). We evaluate a set of baseline learning strategies on these tasks, and show that a novel model incorporating predictive lookahead is a promising approach for learning from a teacher's response. In particular, a surprising result is that it can learn to answer questions correctly without any reward-based supervision at all.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, background): “Dialog based learning (Weston, 2016; Li et al., 2016; Hancock et al., 2019) develops techniques to learn from interactive natural language feedback in dialog.”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “There has also been some work on predicting conversation quality from signals such as sentiment or questions (Ghandeharioun et al., 2019) or using other forms of feedback from the conversation (Li et al., 2017a,b; Weston, 2016; Yang et al., 2017), but results are still preliminary.”
Learning from Dialogue after Deployment: Feed Yourself, Chatbot! (2019, background): “The specific case of learning actively from dialogue during deployment was explored for the question answering (QA) setting in (Weston, 2016) and (Li et al.”
Training Classifiers with Natural Language Explanations (2018, background): “Weston (2016); Li et al. (2016) learn from natural language feedback in a dialogue.”
Learning Through Dialogue Interactions (2016, methodology): “Offline Supervised Settings: The first learning strategy we adopt is the reward-based imitation strategy (denoted vanilla-MemN2N) described in (Weston, 2016), where at training time, the model maximizes the log likelihood probability of the correct answers the student gave (examples with incorrect f”
Training Language Models with Language Feedback (2022, background): “A few prior works also learn from language feedback, for the purpose of ranking candidate outputs rather than generating outputs (Weston, 2016; Li et al., 2016; Hancock et al., 2019; Li et al., 2022).”
Recipes for Building an Open-Domain Chatbot (2020, background): “We note that these models cannot be taught a concept through further conversation, so as-is they will always be stunted, see (Weston, 2016; Hancock et al., 2019) for early work in this direction.”
A Survey of Available Corpora For Building Data-Driven Dialogue Systems: The Journal Version (2015, background): “These models have demonstrated promising results for several non-goal-driven dialogue tasks (Ritter et al., 2011; Vinyals and Le, 2015; Lowe et al., 2015a; Sordoni et al., 2015b; Shang et al., 2015; Li et al., 2015; Serban et al., 2016; Serban et al., 2017b,a; Dodge et al., 2015; Williams and Zweig,”
Dialogue Learning With Human-In-The-Loop (2016, positive): “The difference between our simulation and the original ﬁxed tasks of Weston (2016) is that models are trained on-the-ﬂy.”
Generative Deep Neural Networks for Dialogue: A Short Review (2016, methodology): “In another line of work, researchers have started proposing alternative training and response selection criteria [Weston, 2016].”
Tracking the World State with Recurrent Entity Networks (2016, methodology): “Recent works have made some progress towards the second goalof f rward modeling, for instance in capturing simple physics (Lerer et al., 2016), predicting future frames in video (Mathieu et al., 2015) or responses in dialog (Weston, 2016).”
A Neural Knowledge Language Model (2016, background): “In particular, for applications such as question answering [21, 42, 8] and dialogue modeling [40, 34, 41], these words are of our main interest.”
Why Build an Assistant in Minecraft? (2019, background): “, “Yes, that’s right!”) was explored for the question answering (QA) setting in [92] and [48], and in a chit-chat setting in [33].”
MnnFast: A Fast and Scalable System Architecture for Memory-Augmented Neural Networks (2019, background): “Nowadays, thanks to its huge reasoning power, MemNN is widely used in various fields from simple dialog comprehension to question & answering system using a large-scale dataset (e.g., Wikipedia) [17, 32, 49, 69, 76] Figure 2 shows the high-level overview of MemNN’s computational structure.”
Continuous and Interactive Factual Knowledge Learning in Veriﬁcation Dialogues (2020, background): “Among dialogue systems [48, 25, 26, 10, 53, 57], the work in [26] allows the learner to ask questions.”
Teaching Machines to Describe Images via Natural Language Feedback (2017, background): “is slowly finding its way into everyone’s lives, be in the form of social bots [36, 2], personal assistants [24, 13, 32], or household robots [1], it becomes critical to allow non-expert users to teach and guide their robots [37, 18].”
Training Language Models with Language Feedback at Scale (2023, background): “A few prior works also learn from language feedback for the purpose of ranking candidate outputs rather than generating outputs (Weston, 2016; Li et al., 2016; Hancock et al., 2019; Li et al., 2022; Xu et al., 2022).”
Language Models are Unsupervised Multitask Learners (2019, background): “While it is a large step from the well-posed setup described above to the messiness of “language in the wild”, Weston (2016) argues, in the context of dialog, for the need to develop systems capable of learning from natural language directly and demonstrated a proof of concept – learning a QA task…”
Towards a Continuous Knowledge Learning Engine for Chatbots (2018, background): “Also, no existing dialogue systems (Vinyals and Le, 2015; Li et al., 2016a; Bordes and Weston, 2016; Weston, 2016; Zhang et al., 2017) employ lifelong learning to train prediction models by using information/knowledge retained in the past.”