A Multi-Type Multi-Span Network for Reading Comprehension that Requires Discrete Reasoning

By Minghao Hu, Yuxing Peng, Zhen Huang, Dongsheng Li. 2019

98 citations recorded by the source corpus.

Rapid progress has been made in the field of reading comprehension and question answering, where several systems have achieved human parity in some simplified settings. However, the performance of these models degrades significantly when they are applied to more realistic scenarios, such as answers involve various types, multiple text strings are correct answers, or discrete reasoning abilities are required. In this paper, we introduce the Multi-Type Multi-Span Network (MTMSN), a neural reading comprehension model that combines a multi-type answer predictor designed to support various answer types (e.g., span, count, negation, and arithmetic expression) with a multi-span extraction method for dynamically producing one or multiple text spans. In addition, an arithmetic expression reranking mechanism is proposed to rank expression candidates for further confirming the prediction. Experiments show that our model achieves 79.9 F1 on the DROP hidden test set, creating new state-of-the-art results. Source code (https://github.com/huminghao16/MTMSN) is released to facilitate future work.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Chain of Thought Prompting Elicits Reasoning in Large Language Models (2022, neutral): “, 2020), b: MTMSN (Hu et al., 2019), c: relevance classifier (Roy & Roth, 2015), d: GPT-2 finetuned with k-fold CV (Lan et al.”
Reasoning Like Program Executors (2022, methodology): “to perform binary classification on concatenated 456 question-option-context pairs, as suggested in Liu 457 et al. (2020). (vi) On SVAMP, the encoder-only 458 model is not suitable since the answers are out-of459 context.”
Benefits of Intermediate Annotations in Reading Comprehension (2020, methodology): “However, models trained on these datasets (Hu et al., 2019; Andor et al., 2019) only have the ﬁnal answer as supervision, leaving the model guessing at the correct latent reasoning.”
Injecting Numerical Reasoning Skills into Language Models (2020, methodology): “Third, MTMSNBASE substantially outperforms GENBERT on questions whose answer is a list of non-contiguous spans.”
Evaluating Models’ Local Decision Boundaries via Contrast Sets (2020, methodology): “…involve ﬁne-tuning a pretrained language model (e.g., ELMo (Peters et al., 2018), BERT (Devlin et al., 2019), R O BERT A (Liu et al., 2019), XLNet (Yang et al., 2019), etc.) or applying a task-speciﬁc architecture on top of one (e.g., Hu et al. (2019) add a DROP-speciﬁc model on top of BERT).”
Neural Module Networks for Reasoning over Text (2019, methodology): “The resulting model achieves a score of 65.4 F1 on the complete validation data of DROP, as compared to MTMSN that achieves 72.8 F1.”
Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models (2021, methodology): “While a constrained-decoding approach (Hokamp and Liu, 2017; Hu et al., 2019a) could be used here to further promote the use of the vocabulary hints, this simple approach was effective and more generally applicable to other hints in our use-case.”
Evaluating NLP Models via Contrast Sets (2020, methodology): “For each subset, we evaluate MTMSN’s performance and show the results in the Table below.”
Question Directed Graph Attention Network for Numerical Reasoning over Text (2020, methodology): “Some recent works (LCGN (Hu et al., 2019b), NMNs (Gupta et al., 2020), NumNet (Ran et al., 2019)) attaching reasoning capabilities to models shows a promising direction.”
Successive Prompting for Decomposing Complex Questions (2022, methodology): “State-of-the-art models on DROP, however, use a combination of BERT-based contextual models along with a calculator that performs discrete operations (Andor et al., 2019; Segal et al., 2020; Hu et al., 2019).”
Making Language Models Better Reasoners with Step-Aware Verifier (2022, background): “…systems, including knowledge graphs (Mihaylov and Frank, 2018; Bauer et al., 2018; Kundu et al., 2019; Wang et al., 2019; Lin et al., 2019; Ding et al., 2019; Feng et al., 2020; Wang et al., 2022b), or question taxonomies (Dua et al., 2019; Andor et al., 2019; Hu et al., 2019b; Wang et al., 2022a).”
On the Advance of Making Language Models Better Reasoners (2022, background): “…systems, including knowledge graphs (Mihaylov and Frank, 2018; Bauer et al., 2018; Kundu et al., 2019; Wang et al., 2019; Lin et al., 2019; Ding et al., 2019; Feng et al., 2020; Wang et al., 2022b), or question taxonomies (Dua et al., 2019; Andor et al., 2019; Hu et al., 2019b; Wang et al., 2022a).”
OPERA: Operation-Pivoted Discrete Reasoning over Text (2022, methodology): “Based on NAQANET, MTMSN (Hu et al., 2019) adds a negation predictor to solve the negative question and re-rank arithmetic expression candidates.”
Obtaining Faithful Interpretations from Compositional Neural Networks (2020, background): “Both models that assume an intermediate structure (Andreas et al., 2016; Jiang and Bansal, 2019) and models without such structure (Tan and Bansal, 2019; Hu et al., 2019; Min et al., 2019) have been proposed for these reasoning problems.”
Coreferential Reasoning Learning for Language Representation (2020, methodology): “For QUOREF, we further employ two extra components to process multiple mentions of the answers: (1) Spurred by the idea from Hu et al. (2019) in handling multiple answer spans problem, we utilize the representation of [CLS] to predict the number of answers.”