Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies

By Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, Jonathan Berant. 2021

1,119 citations recorded by the source corpus.

Abstract A key limitation in current datasets for multi-hop reasoning is that the required steps for answering the question are mentioned in it explicitly. In this work, we introduce StrategyQA, a question answering (QA) benchmark where the required reasoning steps are implicit in the question, and should be inferred using a strategy. A fundamental challenge in this setup is how to elicit such creative questions from crowdsourcing workers, while covering a broad range of potential strategies. We propose a data collection procedure that combines term-based priming to inspire annotators, careful control over the annotator population, and adversarial filtering for eliminating reasoning shortcuts. Moreover, we annotate each question with (1) a decomposition into reasoning steps for answering it, and (2) Wikipedia paragraphs that contain the answers to each step. Overall, StrategyQA includes 2,780 examples, each consisting of a strategy question, its decomposition, and evidence paragraphs. Analysis shows that questions in StrategyQA are short, topic-diverse, and cover a wide range of strategies. Empirically, we show that humans perform well (87%) on this task, while our best baseline reaches an accuracy of ∼ 66%.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, neutral): “…evaluate PaLM’s performance on the arithmetic datasets GSM8K (Cobbe et al., 2021), SVAMP (Patel et al., 2021), MAWPS (Koncel-Kedziorski et al., 2016), and AQuA (Ling et al., 2017), as well as the commonsense reasoning datasets CommonsenseQA (Talmor et al., 2019) and StrategyQA (Geva et al., 2021).”
Chain of Thought Prompting Elicits Reasoning in Large Language Models (2022, neutral): “We ﬁnd that with the same prompts, chain-of-thought prompting improves performance across all three models (LaMDA, GPT-3, and PaLM) for all datasets except CSQA and StrategyQA for GPT-3 (Table 1, Table 4, Table 5).”
Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022, methodology): “In addition, we observed that language models can sometimes generate incorrect or nonsensical reasoning paths (e.g., the StrategyQA example in Table 4, the two population numbers are not exactly correct), and further work is needed to better ground models’ rationale generations.”
Scaling Instruction-Finetuned Language Models (2022, background): “, 2021), multi-hop reasoning (Geva et al., 2021), and natural language inference (Camburu et al.”
Internet-augmented language models through few-shot prompting for open-domain question answering (2022, background): “, 2018) and STRATEGYQA (Geva et al., 2021); and a fact-checking dataset: the single-hop FEVER (Thorne et al.”
Large Language Models Can Self-Improve (2022, neutral): “, 2021) are arithmetic reasoning tasks; (2) StrategyQA (Geva et al., 2021) is a commonsense reasoning task; (3) ANLI-A1 (Mihaylov et al.”
Large Language Models are Zero-Shot Reasoners (2022, methodology): “StrategyQA requires models to infer an implicit multi-hop reasoning to answer questions [Geva et al., 2021].”
CommonsenseQA 2.0: Exposing the Limits of AI through Gamification (2021, neutral): “Examples for this include 368 [39, 40, 41, 12, 42, 34, 43, 38, 44], with recent works featuring re-training during dataset collection 369 (Beat-the-AI [22], StrategyQA [23] and Dynabench [21]).”
Least-to-Most Prompting Enables Complex Reasoning in Large Language Models (2022, background): “For instance, a prompt that demonstrates decomposing math word problems (as seen in Table 9) isn’t effective for teaching large language models to break down common sense reasoning problems, such as “Did Aristotle use a laptop?” (Geva et al., 2021).”
GLM-130B: An Open Bilingual Pre-trained Model (2022, methodology): “, 2019) and StrategyQA (Geva et al., 2021) as the evaluation datasets for CBQA.”
PaLM 2 Technical Report (2023, background): “, 2019), StrategyQA (Geva et al., 2021), CommonsenseQA (CSQA; Talmor et al.”
Transcending Scaling Laws with 0.1% Extra Compute (2022, neutral): “[Geva et al., 2021] and CommonsenseQA [Talmor et al., 2019] benchmarks.”
Explaining Answers with Entailment Trees (2021, neutral): “C L ] (Xie et al., 2020) 2 (Jhamtani and Clark, 2020) 3 (Yang et al., 2018) 4 (Inoue et al., 2020) 5 (Geva et al., 2021) Table 1: A comparison of E NTAILMENT B ANK with other similar datasets.”
Teaching Small Language Models to Reason (2022, neutral): “This work explores CoT knowledge distillation (Hinton et al., 2015) from PaLM 540B (Chowdhery et al., 2022) and GPT-3 175B (Brown et al., 2020) to different sizes of the smaller language model T5 (Raffel et al., 2020), such as T5 XXL, XL and base, which have 11 billion, 3 billion and 220 million…”
The Unreliability of Explanations in Few-Shot In-Context Learning (2022, background): “In fact, in Wei et al. (2022) and Chowdhery et al. (2022), explanations only show mild beneﬁt on open-domain QA tasks like StrategyQA (Geva et al., 2021) that are closer to our setting.”
Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations (2022, methodology): “100 provide experiments on StrategyQA (Geva et al., 2021), to evaluate the generalizability of M AIEU - TIC PROMPTING in multi-hop setting.”
Competition-Level Problems are Effective LLM Evaluators (2023, background): “Researchers have proposed many benchmarks requiring various reasoning skills, including commonsense reasoning (Talmor et al., 2018; Geva et al., 2021), numerical reasoning (Dua et al., 2019), multi-hop reasoning (Yang et al., 2018), arithmetic reasoning (Patel et al., 2021; Cobbe et al., 2021),…”