A Unified Encoder-Decoder Framework with Entity Memory

By Zhihan Zhang, W. Yu, Chenguang Zhu, Meng Jiang. 2022

12 citations recorded by the source corpus.

Entities, as important carriers of real-world knowledge, play a key role in many NLP tasks.We focus on incorporating entity knowledge into an encoder-decoder framework for informative text generation. Existing approaches tried to index, retrieve, and read external documents as evidence, but they suffered from a large computational overhead. In this work, we propose an encoder-decoder framework with an entity memory, namely EDMem. The entity knowledge is stored in the memory as latent representations, and the memory is pre-trained on Wikipedia along with encoder-decoder parameters. To precisely generate entity names, we design three decoding methods to constrain entity generation by linking entities in the memory. EDMem is a unified framework that can be used on various entity-intensive question answering and generation tasks. Extensive experimental results show that EDMem outperforms both memory-based auto-encoder models and non-memory encoder-decoder models.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs (2022, methodology): “…language models, but we focus on accessing GPT-3 (Brown et al., 2020) through an API, since we do not have access to some of the larger models out there (PaLM; Chowdhery et al., 2022) nor do we have the computational resources to run an open-source version of GPT-3 (OPT; Zhang et al., 2022b).”