WebGPT: Browser-assisted question-answering with human feedback
By Reiichiro Nakano, Jacob Hilton, S. Balaji, Jeff Wu, Ouyang Long, Ouyang Long, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju and 9 more. 2021
1,953 citations recorded by the source corpus.
We fine-tune GPT-3 to answer long-form questions using a text-based web-browsing environment, which allows the model to search and navigate the web. By setting up the task so that it can be performed by humans, we are able to train models on the task using imitation learning, and then optimize answer quality with human feedback. To make human evaluation of factual accuracy easier, models must collect references while browsing in support of their answers. We train and evaluate our models on ELI5, a dataset of questions asked by Reddit users. Our best model is obtained by fine-tuning GPT-3 using behavior cloning, and then performing rejection sampling against a reward model trained to predict human preferences. This model's answers are preferred by humans 56% of the time to those of our human demonstrators, and 69% of the time to the highest-voted answer from Reddit.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Retrieval-Augmented Generation, Large Language Models
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Percy Liang
- Yejin Choi
- Graham Neubig
- Quoc V. Le
- Yoshua Bengio
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Andrew Zisserman
- Cordelia Schmid
In Retrieval-Augmented Generation
- Sebastian Riedel
- Danqi Chen
- Hannaneh Hajishirzi
- Luyu Gao
- Wen-tau Yih
- Jamie Callan
- Jason Aaron Edward Weston
- Matei A. Zaharia
In Large Language Models
- Luke S. Zettlemoyer
- Hinrich Schütze
- Yejin Choi
- Timo Schick
- Jianfeng Gao
- Hannaneh Hajishirzi
- Samuel R. Bowman
- Noah A. Smith
Citations in the corpus
- Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, background): “In Table 6, we report the statistics of reward modeling data that we collected over time, and present them against multiple open-source preference datasets including Anthropic Helpful and Harmless (Bai et al., 2022a), OpenAI Summarize (Stiennon et al., 2020), OpenAI WebGPT (Nakano et al., 2021), Sta”
- Few-shot Learning with Retrieval Augmented Language Models (2022, background): “Recently, different works have proposed to train large language models to interact with a search engine, by generating text queries, and using the retrieved documents as additional context (Nakano et al., 2021; Thoppilan et al., 2022; Shuster et al., 2022).”
- ReAct: Synergizing Reasoning and Acting in Language Models (2022, methodology): “(c) Acting-only prompt (Act), which removes thoughts in ReAct trajectories, loosely resembling how WebGPT (Nakano et al., 2021) interacts with the Internet to answer questions, though it operates on a different task and action space, and uses imitation and reinforcement learning instead of prompting”
- Toolformer: Language Models Can Teach Themselves to Use Tools (2023, background): “…a tool in an interactive way – especially for tools such as search engines, that could potentially return hundreds of different results, enabling a LM to browse through these results or to refine its search query in a similar spirit to Nakano et al. (2021) can be crucial for certain applications.”
- TruthfulQA: Measuring How Models Mimic Human Falsehoods (2021, methodology): “Appendix B.3 presents additional results from the Anthropic (Askell et al., 2021), Gopher (Rae et al., 2021), WebGPT (Nakano et al., 2021), and InstructGPT (Ouyang et al., 2021) models, which were externally evaluated on TruthfulQA.”
- OPT: Open Pre-trained Transformer Language Models (2022, background): “We aim to be fully accountable for the development lifecycle of OPT-175B, and only through increasing transparency around LLM development can we start understanding the limitations and risks of LLMs before broader deployment occurs.”
- Augmented Language Models: a Survey (2023, methodology): “A successful example of RLHF used to teach a LM to use an external tool stems from WebGPT (Nakano et al., 2021), discussed in 3.”
- The Flan Collection: Designing Data and Methods for Effective Instruction Tuning (2023, background): “, 2022), and (b) offering human feedback signals on model responses (Ouyang et al., 2022; Glaese et al., 2022; Bai et al., 2022a; Nakano et al., 2021; Bai et al., 2022b).”
- Autoregressive Search Engines: Generating Substrings as Document Identifiers (2022, background): “…Google” Recently, language models have been shown to be able to directly generate search queries for modern web search engines either with finetuning on demonstrations (Komeili et al., 2021; Shuster et al., 2022) and human preferences (Nakano et al., 2021) or via prompting (Lazaridou et al., 2022).”
- Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion (2022, background): “WebGPT (Nakano et al., 2021) also applies internet search to QA tasks, as does the work of Lazaridou et al. (2022); neither applies to dialogue or general LM tasks, and neither work is openly available.”
- Internet-augmented language models through few-shot prompting for open-domain question answering (2022, background): “1, the complexity of certain (multi-hop) questions pose problems to this simple approach, calling for more sophisticated learning to search approaches (Adolphs et al., 2021; Komeili et al., 2021; Nakano et al., 2021).”
- The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpus (2021, background): “…relies on commercial, black-box search engines, focuses on individual tasks, primarily ODQA (Joshi et al., 2017; Bajaj et al., 2018; Talmor and Berant, 2019; Nakano et al., 2021), or only uses general web content at pre-training time (Guu et al., 2020; Borgeaud et al., 2022; Lewis et al., 2020a).”
- Scaling Laws for Reward Model Overoptimization (2022, methodology): “…to train language models to perform a variety of complex tasks that are hard to judge automatically, including summarization [Stiennon et al., 2020], question-answering [Nakano et al., 2021, Menick et al., 2022], and general assistance [Ouyang et al., 2022, Bai et al., 2022, Glaese et al., 2022].”
- Improving alignment of dialogue agents via targeted human judgements (2022, methodology): “, 2022), WebGPT (Nakano et al., 2021), and BlenderBot 3 (Shuster et al.”
- Training Language Models with Language Feedback (2022, background): “highly according to human preferences, or a predictive model thereof (Ziegler et al., 2019; Stiennon et al., 2020; Nakano et al., 2021; Ouyang et al., 2022).”