Adaptive Semiparametric Language Models

By Dani Yogatama, Cyprien de Masson d'Autume, Lingpeng Kong. 2021

107 citations recorded by the source corpus.

Abstract We present a language model that combines a large parametric neural network (i.e., a transformer) with a non-parametric episodic memory component in an integrated architecture. Our model uses extended short-term context by caching local hidden states—similar to transformer-XL—and global long-term memory by retrieving a set of nearest neighbor tokens at each timestep. We design a gating function to adaptively combine multiple information sources to make a prediction. This mechanism allows the model to use either local context, short-term memory, or long-term memory (or any combination of them) on an ad hoc basis depending on the context. Experiments on word-based and character-based language modeling datasets demonstrate the efficacy of our proposed method compared to strong baselines.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Few-shot Learning with Retrieval Augmented Language Models (2022, background): “In addition to their memorisation abilities, such architectures are attractive due to a number of other established advantages in terms of adaptability, interpretability and eﬃciency (Guu et al., 2020; Lewis et al., 2020; Yogatama et al., 2021; Borgeaud et al., 2021, inter alia).”
Improving language models by retrieving from trillions of tokens (2021, background): “Spalm (Yogatama et al., 2021) addresses this limitation by adding an extra gating network to post-process the retrieved data; yet most of the network is unaffected by the retrieval during inference.”
Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion (2022, background): “In the language modeling space, there is a large body of work on nearest neighbor and cache-based language modeling (Khandelwal et al., 2020; Grave et al., 2017; Merity et al., 2017; Khandelwal et al., 2021; Yogatama et al., 2021) for accessing a large set of documents.”
Internet-augmented language models through few-shot prompting for open-domain question answering (2022, background): “language models have been recently gaining momentum (Khandelwal et al., 2019; Guu et al., 2020; Yogatama et al., 2021; Borgeaud et al., 2021), extending monolithic parametric models with information from a knowledge source.”
You Only Need One Model for Open-domain Question Answering (2021, methodology): “Neural Retriever Augmented Language Modeling (NRALM): Augmenting language models with neural retrieval has been shown to be very effective, such as by retrieving nearest neighbor words for LM tasks (Khandelwal et al., 2020; Yogatama et al., 2021) or Machine Translation (Khandelwal et al.”
End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering (2021, background): “In left-to-right language modeling, retrieving similar words from an external memory has been shown to improve perplexity (Khandelwal et al., 2020, Yogatama et al., 2021).”
Retrieval Augmentation Reduces Hallucination in Conversation (2021, methodology): “…includes a retrievalaugmentation step includes the area of language modeling, where it is used for pre-training (Guu et al., 2020), and as a memory (Yogatama et al., 2021), especially using k-nearest neighbor-based cache models (Khandelwal et al., 2021, 2020; Grave et al., 2016; Merity et al.,…”
Mind the Gap: Assessing Temporal Generalization in Neural Language Models (2021, background): “Furthermore, recent semi-parametric models (Guu et al., 2020; Lewis et al., 2020b; Karpukhin et al., 2020; Khandelwal et al., 2020; Yogatama et al., 2021) lend themselves well to continual learning, where new knowledge can be stored in an external memory, which can be updated without retraining the…”
Towards mental time travel: a hierarchical memory for reinforcement learning agents (2021, background): “Several recent papers have proposed systems that can be interpreted as episodic memories from this perspective, specifically looking up related contexts from the training corpus using either nearest neighbors [25], or attention [61].”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, background): “[315] incorporate a gating mechanism to replace the fixed interpolation parameter in kNN-LM, which adaptively combines short-term memory with long-term memory (i.”
Large language models could change the future of behavioral healthcare: a proposal for responsible development and evaluation (2024, background): “Inaddition,itisalreadypossibletoaugmentthememoryofLLMswith “ vector databases, ” which would have the added bene ﬁ t of retaining inspectable learnings and summaries across clinical encounters 72 .”
Memorizing Transformers (2022, background): “Yogatama et al. (2021) extend this approach by a gating mechanism and a process to compress the context into keys for retrieval.”
\infty-former: Infinite Memory Transformer (2021, background): “More recently, Khandelwal et al. (2019) and Yogatama et al. (2021) proposed using nearest neighbors to retrieve words from a key-based memory constructed based on the training data.”
LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks (2022, background): “[101] Dani Yogatama, Cyprien de Masson d’Autume, and Lingpeng Kong.”
A Review on Language Models as Knowledge Bases (2022, background): “Episodic non-parametric memory can also be introduced to LMs to remember speciﬁc knowledge, including both short-term and long-term contexts (de Masson d’Autume et al., 2019; Yogatama et al., 2021; Liu et al., 2022).”
Relational Memory-Augmented Language Models (2022, methodology): “• We study the relationship between our method and a typical memory-augmented language model which stores word tokens in its memory (Yogatama et al., 2021).”
Pitfalls of Static Language Modelling (2021, background): “Another promising direction is to disentangle the acquisition of up-to-date knowledge (for instance by retrieving external information) from the language learning itself, as recently proposed by Guu et al. (2020) and Yogatama et al. (2021), among others.”