SG-Net: Syntax-Guided Machine Reading Comprehension

By Zhuosheng Zhang, Yuwei Wu, Junru Zhou, Sufeng Duan, Hai Zhao, Rui Wang. 2019

195 citations recorded by the source corpus.

For machine reading comprehension, the capacity of effectively modeling the linguistic knowledge from the detail-riddled and lengthy passages and getting ride of the noises is essential to improve its performance. Traditional attentive models attend to all words without explicit constraint, which results in inaccurate concentration on some dispensable words. In this work, we propose using syntax to guide the text modeling by incorporating explicit syntactic constraints into attention mechanism for better linguistically motivated word representations. In detail, for self-attention network (SAN) sponsored Transformer-based encoder, we introduce syntactic dependency of interest (SDOI) design into the SAN to form an SDOI-SAN with syntax-guided self-attention. Syntax-guided network (SG-Net) is then composed of this extra SDOI-SAN and the SAN from the original Transformer encoder through a dual contextual architecture for better linguistics inspired representation. To verify its effectiveness, the proposed SG-Net is applied to typical pre-trained language model BERT which is right based on a Transformer encoder. Extensive experiments on popular benchmarks including SQuAD 2.0 and RACE show that the proposed SG-Net design helps achieve substantial performance improvement over strong baselines.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Dual Co-Matching Network for Multi-choice Reading Comprehension (2019, background): “2019) but at the same time it is time-consuming and resource-demanding to impart massive amounts of general knowledge from external corpora into a deep language model via pre-training (Sun et al. 2019; Zhang et al. 2019b).”
Neural Module Networks for Reasoning over Text (2019, background): “Recent models have performed well on certain QA datasets, sometimes rivaling humans (Zhang et al., 2019), but it has become increasingly clear that they primarily exploit surface level lexical cues (Jia & Liang, 2017; Feng et al.”
Graphspeech: Syntax-Aware Graph Attention Network for Neural Speech Synthesis (2020, background): “While SAN-encoder builds the long-range dependency between any two tokens to model the global context, it doesn’t model the complex syntactic dependency of words within a sentence [25, 26], which is usually described by a tree structure or a graph.”
Retrospective Reader for Machine Reading Comprehension (2020, methodology): “…side 1 of MRC models [ Hu et al. , 2019; Back et al. , 2020 ] , though it has been shown that better decoder or better manner of using encoder still has a signiﬁcant impact on MRC performance, no matter how strong the encoder (i.e., the adopted pre-trained LM) it is [ Zhang et al. , 2020a ] .”
RikiNet: Reading Wikipedia Pages for Natural Question Answering (2020, background): “…of large-scale pre-trained language models (Devlin et al., 2018), state-of-the-art MRC models (Ju et al., 2019; Yang et al., 2019; Lan et al., 2019; Zhang et al., 2019; Liu et al., 2019) have already surpassed human-level performance on certain commonly used MRC benchmark datasets, such as SQuAD…”
Semantics-aware BERT for Language Understanding (2019, background): “To this end, there is a recent trend of incorporating extra knowledge to pre-trained language models (Zhang et al. 2020b).”