Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion
By Kurt Shuster, Mojtaba Komeili, Leonard Adolphs, Stephen Roller, Arthur Szlam, Jason Aaron Edward Weston. 2022
150 citations recorded by the source corpus.
Language models (LMs) have recently been shown to generate more factual responses by employing modularity (Zhou et al., 2021) in combination with retrieval (Adolphs et al., 2021). We extend the recent approach of Adolphs et al. (2021) to include internet search as a module. Our SeeKeR (Search engine->Knowledge->Response) method thus applies a single LM to three modular tasks in succession: search, generating knowledge, and generating a final response. We show that, when using SeeKeR as a dialogue model, it outperforms the state-of-the-art model BlenderBot 2 (Chen et al., 2021) on open-domain knowledge-grounded conversations for the same number of parameters, in terms of consistency, knowledge and per-turn engagingness. SeeKeR applied to topical prompt completions as a standard language model outperforms GPT2 (Radford et al., 2019) and GPT3 (Brown et al., 2020) in terms of factuality and topicality, despite GPT3 being a vastly larger model. Our code and models are made publicly available.
Subfields: Artificial Intelligence, Information Systems, Retrieval-Augmented Generation, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Luke S. Zettlemoyer
- Mohit Bansal
- Jason Aaron Edward Weston
- Jianfeng Gao
- Percy Liang
- Yejin Choi
- Graham Neubig
- Hannaneh Hajishirzi
In Information Systems
- Sebastian Riedel
- Danqi Chen
- Wen-tau Yih
- Jason Aaron Edward Weston
- Luke S. Zettlemoyer
- Fabio Petroni
- Timo Schick
- Patrick Lewis
In Retrieval-Augmented Generation
- Sebastian Riedel
- Jason Aaron Edward Weston
- Danqi Chen
- Wen-tau Yih
- Arthur Szlam
- Kurt Shuster
- Fabio Petroni
- Patrick Lewis
In Natural Language Processing
- Jason Aaron Edward Weston
- Luke S. Zettlemoyer
- Mohit Bansal
- Sebastian Riedel
- Jianfeng Gao
- Kurt Shuster
- Graham Neubig
- Douwe Kiela
Citations in the corpus
- Few-shot Learning with Retrieval Augmented Language Models (2022, background): “Recently, different works have proposed to train large language models to interact with a search engine, by generating text queries, and using the retrieved documents as additional context (Nakano et al., 2021; Thoppilan et al., 2022; Shuster et al., 2022).”
- OPT: Open Pre-trained Transformer Language Models (2022, background): “We also compare against the fine-tuned R2C2 BlenderBot, a 2.7B parameter BlenderBot-like model trained by Shuster et al. (2022).”
- Autoregressive Search Engines: Generating Substrings as Document Identifiers (2022, background): “…Google” Recently, language models have been shown to be able to directly generate search queries for modern web search engines either with finetuning on demonstrations (Komeili et al., 2021; Shuster et al., 2022) and human preferences (Nakano et al., 2021) or via prompting (Lazaridou et al., 2022).”
- When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels (2022, background): “To evaluate J UICER on state-of-the-art chatbots in such a setting, we thus construct a new sparse sampled version of the existing FITS dataset from Xu et al. (2022), which consists of fully annotated human-model conversations between users and existing state-of-the-art internet-augmented models suc”
- Improving alignment of dialogue agents via targeted human judgements (2022, methodology): “This mechanism also allows for temporal generalisation beyond a static parametric model (Borgeaud et al., 2022; Lewis et al., 2020; Liška et al., 2022; Shuster et al., 2022a).”
- BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage (2022, methodology): “This overall setup is built upon our group’s previous works K2R (Adolphs et al., 2021) and SeeKeR (Shuster et al., 2022), in addition to its predecessors BB1 (Roller et al., 2021) and BB2 (Komeili et al., 2022; Xu et al., 2022a).”
- WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents (2022, background): “Several papers have explored the use of the web for information extraction [34] and retrieval [1], question answering [57, 25], dialog [45], and training language models on webtext [2].”
- Generative Spoken Dialogue Language Modeling (2022, background): “, 2021), and grounding responses on fresh information from the internet (Komeili et al., 2021; Shuster et al., 2022).”
- Effective Theory of Transformers at Initialization (2023, methodology): “[45] – albeit here with shorter warmup and target training lengths – and, for the neural-tangent run, we shift it to (n3/2lr, n−3/2wd) = (20483/2 · 7e-4, 2048−3/2 · 0.”
- Enabling Large Language Models to Generate Text with Citations (2023, background): “Other works prompt or fine-tune LLMs to “retrieve on-the-fly” (Parisi et al., 2022; Schick et al., 2023; Shuster et al., 2022; Jiang et al., 2023; Yao et al., 2023; Press et al., 2022), which offers flexibility of when and what to search.”
- Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback (2022, methodology): “Taking into account modular feedback outperforms using only supervised feedback of final responses in both automatic metric and human evaluations for both BB2 and SeeKeR models.”
- Large language models in medicine (2023, methodology): “Successful validation will involve pragmatic clinical trials demonstrating real benefits with minimized bias and transparent reporting.”