Relational Memory-Augmented Language Models
By Qi Liu, Dani Yogatama, Phil Blunsom. 2022
35 citations recorded by the source corpus.
We present a memory-augmented approach to condition an autoregressive language model on a knowledge graph. We represent the graph as a collection of relation triples and retrieve relevant relations for a given context to improve text generation. Experiments on WikiText-103, WMT19, and enwik8 English datasets demonstrate that our approach produces a better language model in terms of perplexity and bits per character. We also show that relational memory improves coherence, is complementary to token-based memory, and enables causal interventions. Our model provides a simple yet effective way to combine an autoregressive language model and a knowledge graph for more coherent and logical generation.
Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yulia Tsvetkov
- Vidhisha Balachandran
- Antonios Anastasopoulos
- Sachin Kumar
- Lucille Njoo
- Asli Celikyilmaz
- Mona T. Diab
- Badr AlKhamissi
In Natural Language Processing
- Yulia Tsvetkov
- Vidhisha Balachandran
- Antonios Anastasopoulos
- Sachin Kumar
- Lucille Njoo
- Asli Celikyilmaz
- Mona T. Diab
- Badr AlKhamissi
In Model Architectures
- Vidhisha Balachandran
- Yulia Tsvetkov
- Mona T. Diab
- Asli Celikyilmaz
- Badr AlKhamissi
- Millicent Li
- Marjan Ghazvininejad
- Antonios Anastasopoulos
Citations in the corpus
- Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, methodology): “, 2022) Misinformation / Factuality Structured KBs (Wang et al., 2021b; Liu et al., 2022; Yu et al., 2022; Liu et al., 2022; Lewis et al., 2020; de Masson d'Autume et al., 2019; Izacard and Grave, 2021; Hossain et al., 2020; Lewis et al., 2020), Retrieval-based (de Masson d'Autume et al.”
- A Review on Language Models as Knowledge Bases (2022, background): “Episodic non-parametric memory can also be introduced to LMs to remember specific knowledge, including both short-term and long-term contexts (de Masson d’Autume et al., 2019; Yogatama et al., 2021; Liu et al., 2022).”