Internet-augmented language models through few-shot prompting for open-domain question answering

By Angeliki Lazaridou, Elena Gribovskaya, Wojciech Stokowiec, N. Grigorev. 2022

177 citations recorded by the source corpus.

In this work, we aim to capitalize on the unique few-shot capabilities of large-scale language models (LSLMs) to overcome some of their challenges with respect to grounding to factual and up-to-date information. Motivated by semi-parametric language models (LMs), which ground their decisions in external retrieved evidence, we use few-shot prompting to learn to condition LMs on information returned from the web using Google Search, a broad and constantly updated knowledge source. Our approach does not involve fine-tuning or learning additional parameters, thus making it applicable to any LM, offering therefore a strong baseline. Indeed, we find that LMs conditioned on the web surpass performance of closed-book models of similar, or even larger, model sizes in open-domain question answering. Finally, we find that increasing the inference-time compute of models, achieved via using multiple retrieved evidences to generate multiple answers followed by a reranking stage that uses scores generated by the same LMs, leads to better performance and alleviates lower performance of smaller few-shot LMs. All in all, our findings suggest that it might be beneficial to slow down the race towards the biggest model and instead shift attention towards finding more effective ways to use models, including but not limited to, better prompting or increasing inference-time compute.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Retrieval-Augmented Generation, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Jianfeng Gao
  3. Luke S. Zettlemoyer
  4. Percy Liang
  5. Yejin Choi
  6. Hannaneh Hajishirzi
  7. Quoc V. Le
  8. Caiming Xiong

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Mohit Bansal
  3. Trevor Darrell
  4. Dacheng Tao
  5. Jianfeng Gao
  6. Ming-Hsuan Yang
  7. Xiaogang Wang
  8. Jitendra Malik

In Retrieval-Augmented Generation

  1. Sebastian Riedel
  2. Matei A. Zaharia
  3. Christopher Potts
  4. Omar Khattab
  5. Wen-tau Yih
  6. Fabio Petroni
  7. Patrick Lewis
  8. Chenguang Zhu

In Large Language Models

  1. Timo Schick
  2. Hinrich Schütze
  3. Sebastian Riedel
  4. Fabio Petroni
  5. Denny Zhou
  6. Quoc V. Le
  7. Ashish Sabharwal
  8. Wen-tau Yih

Citations in the corpus