Augmenting Transformers with KNN-Based Composite Memory for Dialog

By Angela Fan, Claire Gardent, Chloé Braud, Antoine Bordes. 2020

86 citations recorded by the source corpus.

Various machine learning tasks can benefit from access to external information of different modalities, such as text and images. Recent work has focused on learning architectures with large memories capable of storing this knowledge. We propose augmenting generative Transformer neural networks with KNN-based Information Fetching (KIF) modules. Each KIF module learns a read operation to access fixed external knowledge. We apply these modules to generative dialog modeling, a challenging task where information must be flexibly retrieved and incorporated to maintain the topic and flow of conversation. We demonstrate the effectiveness of our approach by identifying relevant knowledge required for knowledgeable but engaging dialog from Wikipedia, images, and human-written dialog utterances, and show that leveraging this retrieved information improves model performance, measured by automatic and human evaluation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Model Architectures

Citations in the corpus

Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “…have already resulted in significant improvements in many NLP tasks: they have improved factual consistency in summarization models (Huang et al., 2020a), fidelity in translation (Bapna and Firat, 2019), and reliability in dialogue agents (Dinan et al., 2019; Fan et al., 2021; Zhang et al., 2020a).”
End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering (2021, background): “Finally, in dialog modeling, retrieving knowledge-informed text has helped improve factual correctness in the generated conversations (Fan et al., 2021).”
Retrieval Augmentation Reduces Hallucination in Conversation (2021, background): “…et al., 2020); some have shown that model representations themselves can be used nearly outof-the-box for nearest neighbor retrieval of relevant contexts to help in language modeling (Khandelwal et al., 2020), machine translation (Khandelwal et al., 2021), and grounded dialogue (Fan et al., 2021).”
Memorizing Transformers (2022, background): “REALM (Guu et al., 2020), MARGE (Lewis et al., 2020a), RAG (Lewis et al., 2020b), and composite memory for dialog (Fan et al., 2021) retrieve documents from a knowledge base to improve question answering or dialogue.”
Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024, unknown): “For visually grounded dialogue, KNN-based Information Fetching (KIF) [340] enhances generative Transformer for dialog modeling.”
BART for Knowledge Grounded Conversations (2020, methodology): “While some approaches choose to have a separate encoder for knowledge retrieval and for conversation modeling [3, 4], we use a shared encoder to encode the conversation context x , and the filtered knowledge sourceM .”
\infty-former: Infinite Memory Transformer (2021, background): “Similarly, Fan et al. (2021) proposed retrieving sentences from a memory based on the training data and auxiliary information.”
Training Language Models with Memory Augmentation (2022, background): “…de Masson D’Autume et al., 2019; Karpukhin et al., 2020; Guu et al., 2020; Zemlyanskiy et al., 2021; de Jong et al., 2022; Chen et al., 2022; Izacard and Grave, 2021; Singh et al., 2021), dialogue (Fan et al., 2021), and other knowledge-intensive NLP tasks (Lewis et al., 2020; Petroni et al., 2021”
Relational Memory-Augmented Language Models (2022, background): “, 2019; Izacard and Grave, 2021), dialogue (Dinan et al., 2019; Fan et al., 2021; Thulke et al., 2021) and machine translation (Bapna and Firat, 2019; Khandelwal et al.”