What Can a Generative Language Model Answer About a Passage?
By Douglas Summers-Stay, Claire Bonial, Clare R. Voss. 2021
Generative language models trained on large, diverse corpora can answer questions about a passage by generating the most likely continuation of the passage followed by a question/answer pair. However, accuracy rates vary depending on the type of question asked. In this paper we keep the passage fixed, and test with a wide variety of question types, exploring the strengths and weaknesses of the GPT-3 language model. We provide the passage and test questions as a challenge set for other language models.
Subfields: Large Language Models, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Large Language Models
In Natural Language Processing
- Aman Madaan
- Yiming Yang
- Peter Clark
- Niket Tandon
- Ashish Sabharwal
- Tushar Khot
- Mohit Bansal
- Kyle Richardson
Citations in the corpus
- Memory-assisted prompt editing to improve GPT-3 after deployment (2022, positive): “For ERT - CAT and ERT - NL , we found that a higher temperature ( ∼ 0.7) was causing a large divergence in the performance (a difference of ± 10 % accuracy across runs), making reproducibility challenging – similar observations were made by (Summers-Stay et al., 2021).”