A Simple Yet Strong Pipeline for HotpotQA

By Dirk Groeneveld, Tushar Khot, Mausam, Ashish Sabharwal. 2020

42 citations recorded by the source corpus.

State-of-the-art models for multi-hop question answering typically augment large-scale language models like BERT with additional, intuitively useful capabilities such as named entity recognition, graph-based reasoning, and question decomposition. However, does their strong performance on popular multi-hop datasets really justify this added design complexity? Our results suggest that the answer may be no, because even our simple pipeline based on BERT, named Quark, performs surprisingly well. Specifically, on HotpotQA, Quark outperforms these models on both question answering and support identification (and achieves performance very close to a RoBERTa model). Our pipeline has three steps: 1) use BERT to identify potentially relevant sentences independently of each other; 2) feed the set of selected sentences as context into a standard BERT span prediction model to choose an answer; and 3) use the sentence selection model, now with the chosen answer, to produce supporting sentences. The strong performance of Quark resurfaces the importance of carefully exploring simple model designs before using popular benchmarks to justify the value of complex techniques.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Question Answering & Reading Comprehension

Citations in the corpus

Longformer: The Long-Document Transformer (2020, methodology): “The second stage of our two-stage model is based on Groeneveld et al. (2020).”
Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models (2021, methodology): “We also compare MODULARQA with two HotpotQA-specific systems: (1) Quark (Groeneveld et al., 2020), which additionally uses supporting fact annotations to first select relevant sentences and then trains a BERT-based QA system directly on it; and (2) an NMN model (Jiang and Bansal, 2019) specifically ”
♫ MuSiQue: Multihop Questions via Single-hop Question Composition (2021, methodology): “However, this may not be a good enough strategy, because if the subquestions are specific enough, even in an open domain there may not be any good distractor (Groeneveld et al., 2020a).”