Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps

By Xanh Ho, Anh Gia-Tuan Nguyen, Saku Sugawara, Akiko Aizawa. 2020

1,443 citations recorded by the source corpus.

A multi-hop question answering (QA) dataset aims to test reasoning and inference skills by requiring a model to read multiple paragraphs to answer a given question. However, current datasets do not provide a complete explanation for the reasoning process from the question to the answer. Further, previous studies revealed that many examples in existing multi-hop datasets do not require multi-hop reasoning to answer a question. In this study, we present a new multi-hop QA dataset, called 2WikiMultiHopQA, which uses structured and unstructured data. In our dataset, we introduce the evidence information containing a reasoning path for multi-hop questions. The evidence information has two benefits: (i) providing a comprehensive explanation for predictions and (ii) evaluating the reasoning skills of a model. We carefully design a pipeline and a set of templates when generating a question-answer pair that guarantees the multi-hop steps and the quality of the questions. We also exploit the structured format in Wikidata and use logical rules to create questions that are natural but still require multi-hop reasoning. Through experiments, we demonstrate that our dataset is challenging for multi-hop models and it ensures that multi-hop reasoning is required.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning (2022, methodology): “2WikiMultiHop results We evaluated the performance of LLMs on 2WikiMultiHop (Ho et al., 2020) dev subset using exact string match between the generated and the ground truth answers.”
Measuring and Narrowing the Compositionality Gap in Language Models (2022, methodology): “Our prompt has 4 examples, the same ones presented in Table 3 of Ho et al. (2020), in the same order; this demonstrates that we did no prompt engineering or tuning on this dataset.”
Decomposed Prompting: A Modular Approach for Solving Complex Tasks (2022, methodology): “We evaluate our approach on three datasets: (1) 2WikiMultihopQA (Ho et al., 2020) (2) MuSiQue (Trivedi et al.”
Knowledge Graph Prompting for Multi-Document Question Answering (2023, methodology): “To investigate the capability of LLMs for MD-QA, we randomly sample multi-document questions from the development set of 2WikiMQA (Ho et al. 2020) and MuSiQue (Trivedi et al. 2022b), and then prompt LLMs in four different strategies for the answer 1 .”
Efficient Streaming Language Models with Attention Sinks (2023, neutral): “(2020)), and summarization (GovReport (Huang et al., 2021), MultiNews (Fabbri et al., 2019)).”
Search-o1: Agentic Search-Enhanced Large Reasoning Models (2025, neutral): “2WikiMulti-hopQA (2WIKI) (Ho et al., 2020) provides explicit reasoning paths for multi-hop questions.”
Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning (2025, neutral): “(2) Multi-Hop Question Answering : HotpotQA (Yang et al., 2018), 2WikiMultiHopQA (Ho et al., 2020), Musique (Trivedi et al., 2022b), and Bamboogle (Press et al., 2022).”
Shepherd Pre-trained Language Models to Develop a Train of Thought: An Iterative Prompting Approach (2022, methodology): “We conduct experiments on three datasets involving multi-step inference, including two recent multi-hop Question Answering datasets: 2WikiMultiHopQA (Ho et al., 2020) and R4C (Inoue”
Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG (2025, neutral): “• 2WikiMultihopQA: Designed for multihop QA over Wikipedia articles with cross-source integration[62].”
Large Language Models for Information Retrieval: A Survey (2023, neutral): “It is challenging but necessary to consider how to build IR systems free from this category of bias.”
LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (2023, methodology): “, 2018), 2WikiMultihopQA (Ho et al., 2020), and MuSiQue (Trivedi et al.”
Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions (2022, methodology): “, 2018), 2WikiMultihopQA (Ho et al., 2020), MuSiQue (Trivedi et al.”
Gated Delta Networks: Improving Mamba2 with Delta Rule (2024, neutral): “…2018)), scientiﬁc understanding (QasperQA (Dasigi et al., 2021)), multi-hop reasoning (MultiField QA, HotpotQA (Yang et al., 2018), 2WikiMulti QA (Ho et al., 2020), Musique (Trivedi et al., 2022)), document summarization (GovReport (Huang et al., 2021), QMSum (Zhong et al., 2021), MultiNews…”
MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries (2024, background): “Multi-document QA datasets: Question-answering (QA) is a fundamental task in NLP, and several popular benchmarks, such as HotpotQA (Yang et al., 2018), MultiRC (Khashabi et al., 2018), and 2WikiMultiHopQA (Ho et al., 2020), aim to achieve QA from multiple sources of documents.”
Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy (2023, methodology): “We experimented on six datasets of three reasoning tasks: (1) Multi-hop question answering , including HotPotQA (Yang et al., 2018), 2Wiki-MultiHopQA (Ho et al., 2020), MuSiQue (Trivedi et al., 2022b), and Bamboogle (Press et al., 2022).”
Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrieval (2021, background): “These include QAngaroo [27] and 2WikiMulti-HopQA [11], among others.”
MoreHopQA: More Than Multi-hop Reasoning (2024, methodology): “2 Related Work et al., 2020), 2WikiMultihopQA (Ho et al., 2020), and MuSiQue (Trivedi et al., 2022) were introduced, with a greater emphasis on explaining the QA process.”
Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity (2024, methodology): “6) 2WikiMultiHopQA (Ho et al., 2020) is derived from Wikipedia and its associated knowledge graph path, needing 2-hops.”
Active Retrieval Augmented Generation (2023, methodology): “…evaluate FLARE on 4 diverse tasks/datasets involving generating long outputs, including multihop QA (2WikiMul-tihopQA), commonsense reasoning (StrategyQA), long-form QA (ASQA), and open-domain summarization (WikiAsp) (Ho et al., 2020; Geva et al., 2021; Stelmakh et al., 2022; Hayashi et al., 2021).”