Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion

By Kurt Shuster, Mojtaba Komeili, Leonard Adolphs, Stephen Roller, Arthur Szlam, Jason Aaron Edward Weston. 2022

150 citations recorded by the source corpus.

Language models (LMs) have recently been shown to generate more factual responses by employing modularity (Zhou et al., 2021) in combination with retrieval (Adolphs et al., 2021). We extend the recent approach of Adolphs et al. (2021) to include internet search as a module. Our SeeKeR (Search engine->Knowledge->Response) method thus applies a single LM to three modular tasks in succession: search, generating knowledge, and generating a final response. We show that, when using SeeKeR as a dialogue model, it outperforms the state-of-the-art model BlenderBot 2 (Chen et al., 2021) on open-domain knowledge-grounded conversations for the same number of parameters, in terms of consistency, knowledge and per-turn engagingness. SeeKeR applied to topical prompt completions as a standard language model outperforms GPT2 (Radford et al., 2019) and GPT3 (Brown et al., 2020) in terms of factuality and topicality, despite GPT3 being a vastly larger model. Our code and models are made publicly available.

Subfields: Artificial Intelligence, Information Systems, Retrieval-Augmented Generation, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Luke S. Zettlemoyer
  2. Mohit Bansal
  3. Jason Aaron Edward Weston
  4. Jianfeng Gao
  5. Percy Liang
  6. Yejin Choi
  7. Graham Neubig
  8. Hannaneh Hajishirzi

In Information Systems

  1. Sebastian Riedel
  2. Danqi Chen
  3. Wen-tau Yih
  4. Jason Aaron Edward Weston
  5. Luke S. Zettlemoyer
  6. Fabio Petroni
  7. Timo Schick
  8. Patrick Lewis

In Retrieval-Augmented Generation

  1. Sebastian Riedel
  2. Jason Aaron Edward Weston
  3. Danqi Chen
  4. Wen-tau Yih
  5. Arthur Szlam
  6. Kurt Shuster
  7. Fabio Petroni
  8. Patrick Lewis

In Natural Language Processing

  1. Jason Aaron Edward Weston
  2. Luke S. Zettlemoyer
  3. Mohit Bansal
  4. Sebastian Riedel
  5. Jianfeng Gao
  6. Kurt Shuster
  7. Graham Neubig
  8. Douwe Kiela

Citations in the corpus