Quiz-Style Question Generation for News Stories
By Á. Lelkes, Vinh Q. Tran, Cong Yu. 2021
49 citations recorded by the source corpus.
A large majority of American adults get at least some of their news from the Internet. Even though many online news products have the goal of informing their users about the news, they lack scalable and reliable tools for measuring how well they are achieving this goal, and therefore have to resort to noisy proxy metrics (e.g., click-through rates or reading time) to track their performance. As a first step towards measuring news informedness at a scale, we study the problem of quiz-style multiple-choice question generation, which may be used to survey users about their knowledge of recent news. In particular, we formulate the problem as two sequence-to-sequence tasks: question-answer generation (QAG) and distractor, or incorrect answer, generation (DG). We introduce NewsQuizQA, the first dataset intended for quiz-style question-answer generation, containing 20K human written question-answer pairs from 5K news article summaries. Using this dataset, we propose a series of novel techniques for applying large pre-trained Transformer encoder-decoder models, namely PEGASUS and T5, to the tasks of question-answer generation and distractor generation. We show that our models outperform strong baselines using both automated metrics and human raters. We provide a case study of running weekly quizzes on real-world users via the Google Surveys platform over the course of two months. We found that users generally found the automatically generated questions to be educational and enjoyable. Finally, to serve the research community, we are releasing the NewsQuizQA dataset.
Subfields: Artificial Intelligence, Information Systems, Question Answering & Reading Comprehension, Natural Language Generation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Samuel R. Bowman
- Jason Phang
- Mohit Bansal
- Richard Yuanzhe Pang
- He He
- Nikita Nangia
- Angelica Chen
- Yonatan Belinkov
In Information Systems
In Question Answering & Reading Comprehension
Citations in the corpus
- QuALITY: Question Answering with Long Input Texts, Yes! (2021, background): “Most existing datasets (Rajpurkar et al., 2018; Fan et al., 2019; Lelkes et al., 2021) use shorter contexts that humans can read within a few minutes.”