Beyond Goldfish Memory: Long-Term Open-Domain Conversation
By Jing Xu, Arthur Szlam, Jason Aaron Edward Weston. 2021
363 citations recorded by the source corpus.
Despite recent improvements in open-domain dialogue models, state of the art models are trained and evaluated on short conversations with little context. In contrast, the long-term conversation setting has hardly been studied. In this work we collect and release a human-human dataset consisting of multiple chat sessions whereby the speaking partners learn about each other’s interests and discuss the things they have learnt from past sessions. We show how existing models trained on existing datasets perform poorly in this long-term conversation setting in both automatic and human evaluations, and we study long-context models that can perform much better. In particular, we find retrieval-augmented methods and methods with an ability to summarize and recall previous conversations outperform the standard encoder-decoder architectures currently considered state of the art.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Dialogue & Conversational AI
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Luke S. Zettlemoyer
- Jason Aaron Edward Weston
- Mohit Bansal
- Jianfeng Gao
- Percy Liang
- Graham Neubig
- Yejin Choi
- Caiming Xiong
In Computer Vision and Pattern Recognition
- Jason Aaron Edward Weston
- Mohit Bansal
- Luc Van Gool
- Trevor Darrell
- Dacheng Tao
- Xiaogang Wang
- Cordelia Schmid
- Ming-Hsuan Yang
In Natural Language Processing
- Jason Aaron Edward Weston
- Mohit Bansal
- Luke S. Zettlemoyer
- Eric Michael Smith
- Y-Lan Boureau
- Emily Dinan
- Kurt Shuster
- Adina Williams
In Dialogue & Conversational AI
- Kurt Shuster
- Jason Aaron Edward Weston
- Jing Xu
- Andrea Madotto
- Eric Michael Smith
- Stephen Roller
- Mojtaba Komeili
- Caiming Xiong
Citations in the corpus
- OPT: Open Pre-trained Transformer Language Models (2022, methodology): “Furthermore, we evaluated OPT-175B on a subset of the ConvAI2-like MultiSessionChat (MSC) dataset (Xu et al., 2021b) and obtained a perplexity of 9.7 and UF1 of .177, indicating the model is generalizing well across multiple PersonaChat-like datasets.”
- “I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset (2022, methodology): “0 (Komeili et al., 2022; Xu et al., 2022), an encoder/decoder model pre-trained on a Reddit dataset extracted by a third party and made available on pushshift.”
- When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels (2022, methodology): “We fine-tune our models using BlenderBot 2 as the base model (Komeili et al., 2021; Xu et al., 2021).”
- BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage (2022, methodology): “This will require new research, and while we have made steps in this direction (Ju et al., 2022; Xu et al., 2022b), much work remains.”
- Survey of Hallucination in Natural Language Generation (2022, background): “[153] introduce a dataset consisting of over 40 utterances per episode on average.”
- Generative Spoken Dialogue Language Modeling (2022, background): “More recent research focused on utilizing retrieval augmented generation methods (Lewis et al., 2020) for long-context, multi-session conversations (Xu et al., 2021), and grounding responses on fresh information from the internet (Komeili et al., 2021; Shuster et al., 2022).”
- Empirical study on BlenderBot 2.0’s Errors Analysis in terms of Model, Data and User-Centric Approach (2022, methodology): “Thereby, BlenderBot 2.0 (Xu et al., 2021; Komeili et al., 2021), which is an improved version of BlenderBot 1.0 was developed.”
- Evaluating Very Long-Term Conversational Memory of LLM Agents (2024, methodology): “For this, we select a range of initial persona statements p c from the MSC dataset (Xu et al., 2022), each encompassing 4 to 5 sentences.”
- FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation (2022, background): “, 2019), and long-term memory (Xu et al., 2022) are equally important, their evaluation deserves a thorough study on its own.”
- DialFact: A Benchmark for Fact-Checking in Dialogue (2021, background): “False claims and information can be spread by not only humans but also automatic agents as generative models have shown remarkable progress recently (Adiwardana et al., 2020; Xu et al., 2021).”
- Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents (2022, background): “The length of the conversation will also play a role in the performance of models, for example short conversations do not test the ability of models to retain knowledge in the long-term (Xu et al., 2021).”
- Automatically Exposing Problems with Neural Dialog Models (2021, background): “However, even with carefully pre-processed training data from online sources, neural dialog models are prone to issues including generic utterances, repetition, contradiction, and lack of safety (Li et al., 2016a; Welleck et al., 2020; Li et al., 2020; Roller et al., 2021; Xu et al., 2021).”
- PERSONACHATGEN: Generating Personalized Dialogues using GPT-3 (2022, background): “…are: (1) GPT-3 doesn’t get confused between two different per-sonas, so we expect better-quality dialogues (2) GPT-3 can create by adjusting the number of dialogue turns, which is an impactful advantage due to a recent trend when dealing with long-term memory in dialogues (Xu et al., 2021, 2022).”
- MemGPT: Towards LLMs as Operating Systems (2023, background): “I understand that you’re studying computer science and have a keen interest in pretty dynamic areas like Formula 1 and sailing.”
- LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory (2024, neutral): “Early works focused on language modeling evaluation on generating personalized responses from human-human (Xu et al., 2022a) or human-AI (Xu et al., 2022b) chat histories.”