BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
By Christopher Clark, Kenton C. T. Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, Kristina Toutanova. 2019
2,559 citations recorded by the source corpus.
In this paper we study yes/no questions that are naturally occurring — meaning that they are generated in unprompted and unconstrained settings. We build a reading comprehension dataset, BoolQ, of such questions, and show that they are unexpectedly challenging. They often query for complex, non-factoid information, and require difficult entailment-like inference to solve. We also explore the effectiveness of a range of transfer learning baselines. We find that transferring from entailment data is more effective than transferring from paraphrase or extractive QA data, and that it, surprisingly, continues to be very beneficial even when starting from massive pre-trained language models such as BERT. Our best method trains BERT on MultiNLI and then re-trains it on our train set. It achieves 80.4% accuracy compared to 90% accuracy of human annotators (and 62% majority-baseline), leaving a significant gap for future work.
Subfields: Artificial Intelligence, General Social Sciences, Natural Language Processing, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Percy Liang
- Jianfeng Gao
- Yejin Choi
- Hannaneh Hajishirzi
- Noah A. Smith
- Graham Neubig
In General Social Sciences
- Arman Cohan
- Daniel Khashabi
- Yonatan Belinkov
- Tushar Khot
- Luke S. Zettlemoyer
- Ashish Sabharwal
- Matt Gardner
- Dan Roth
In Natural Language Processing
- Mohit Bansal
- Luke S. Zettlemoyer
- Yejin Choi
- Hannaneh Hajishirzi
- Graham Neubig
- Percy Liang
- Noah A. Smith
- Jianfeng Gao
In Evaluation & Benchmarks
- Mohit Bansal
- Yejin Choi
- Hannaneh Hajishirzi
- Luke S. Zettlemoyer
- Samuel R. Bowman
- Matt Gardner
- Ashish Sabharwal
- Greg Durrett
Citations in the corpus
- Training Compute-Optimal Large Language Models (2022, neutral): “• Reading comprehension on RACE (Lai et al., 2017) • Common sense understanding on HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), Wino-grande (Sakaguchi et al., 2020), SIQA (Sap et al., 2019), BoolQ (Clark et al., 2019), and TruthfulQA (Lin et al., 2021).”
- LLaMA: Open and Efficient Foundation Language Models (2023, methodology): “We consider eight standard common sense reasoning benchmarks: BoolQ (Clark et al., 2019), PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), HellaSwag (Zellers et al., 2019), WinoGrande (Sak-aguchi et al., 2021), ARC easy and challenge (Clark et al., 2018) and OpenBookQA (Mihaylov et al., 2018).”
- Mistral 7B (2023, neutral): “…ARC-Challenge [9], CommonsenseQA [24] • World Knowledge (5-shot): NaturalQuestions [16], TriviaQA [15] • Reading Comprehension (0-shot): BoolQ [8], QuAC [7] • Math: GSM8K [10] (8-shot) with maj@8 and MATH [13] (4-shot) with maj@4 • Code: Humaneval [5] (0-shot) and MBPP [2] (3-shot) • Popular…”
- Mixtral of Experts (2024, neutral): “…ARC-Challenge [8], CommonsenseQA [30] • World Knowledge (5-shot): NaturalQuestions [20], TriviaQA [19] • Reading Comprehension (0-shot): BoolQ [7], QuAC [5] • Math: GSM8K [9] (8-shot) with maj@8 and MATH [17] (4-shot) with maj@4 • Code: Humaneval [4] (0-shot) and MBPP [1] (3-shot) • Popular…”
- How Much Knowledge Can You Pack into the Parameters of a Language Model? (2020, background): “…to the model along-side the question (for example, an article that contains the answer (Rajpurkar et al., 2016; Zhang et al., 2018; Khashabi et al., 2018; Clark et al., 2019)) or allows the model to look up informa-a tion in an external knowledge source (Berant et al., 2013; Chen et al., 2017).”
- SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (2019, background): “We collected data to produce conservative estimates for human performance on several tasks that we did not ultimately include in our benchmark, including GAP (Webster et al., 2018), PAWS (Zhang et al., 2019), Quora Insincere Questions, 8 Ultrafine Entity Typing (Choi et al., 2018b), and Empathetic Re”
- Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model (2022, background): “BoolQ BoolQ [10] is a dataset of yes/no questions, with supporting Wikipedia paragraphs to answer them.”
- Efficient Large Scale Language Modeling with Mixtures of Experts (2021, methodology): “We run fine-tuning for a fixed number of epochs (100 for BoolQ, OpenBookQA, StoryCloze, and PIQA, 25 for HellaSwag, Winogrande, and SST-2, 6 for MNLI) and perform model selection based on validation set accuracy.”
- GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021, background): “…(Choi et al., 2018), SQuADv2 (Rajpurkar et al., 2018), RACEh (Lai et al., 2017), RACE-m (Lai et al., 2017) SuperGLUE: (Wang et al., 2019) BoolQ (Clark et al., 2019), CB (de Marneffe et al., 2019), COPA (Gordon et al., 2012), RTE (Dagan et al., 2006), WiC (Pilehvar & Camacho-Collados, 2018),…”
- Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022, methodology): “Here we perform a study using self-consistency to see if it can help fill in the gap, over a set of common NLP tasks, including (1) Closed-Book Question Answering: BoolQ (Clark et al., 2019), HotpotQA (Yang et al.”
- Finetuned Language Models Are Zero-Shot Learners (2021, background): “Reading comprehension datasets: BoolQ, MultiRC, OBQA.”
- Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus (2021, neutral): “• BoolQ (Clark et al., 2019) • CoLA (Warstadt et al., 2019) • MNLI (Williams et al., 2018) • MRPC (Dolan and Brockett, 2005) • QNLI (Rajpurkar et al., 2016; Wang et al., 2019b) • RTE (Dagan et al., 2005; Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009) • SST-2 (Socher et al.,…”
- Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies (2021, background): “• BOOLQ (Clark et al., 2019): A dataset for boolean question answering.”
- TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages (2020, background): “Others have focused exclusively on particular answer types such as Boolean questions (Clark et al., 2019).”
- The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, neutral): “Tasks Type Random small core main ext HellaSwag (Zellers et al., 2019) Sentence completion 25.0 ✓ ✓ ✓ ✓ LAMBADA (Paperno et al., 2016) Sentence completion 0.0 ✓ ✓ ✓ Winogrande (Sakaguchi et al., 2021) Coreference resolution 50.0 ✓ ✓ ✓ ✓ PIQA (Bisk et al., 2020) Multiple-choice question answering 50.”