Words or Characters? Fine-grained Gating for Reading Comprehension

By Zhilin Yang, Bhuwan Dhingra, Ye Yuan, Junjie Hu, William W. Cohen, Ruslan Salakhutdinov. 2016

101 citations recorded by the source corpus.

Previous work combines word-level and character-level representations using concatenation or scalar weighting, which is suboptimal for high-level tasks like reading comprehension. We present a fine-grained gating mechanism to dynamically combine word-level and character-level representations based on properties of the words. We also extend the idea of fine-grained gating to modeling the interaction between questions and paragraphs for reading comprehension. Experiments show that our approach can improve the performance on reading comprehension tasks, achieving new state-of-the-art results on the Children's Book Test dataset. To demonstrate the generality of our gating mechanism, we also show improved results on a social media tag prediction task.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Semi-Supervised QA with Generative Domain-Adaptive Nets (2017, methodology): “Since the answer a is a sequence of consecutive word tokens in the paragraph p , we apply two soft-max layers on top of H Kp to predict the start and end indices of a , following Yang et al. (2017).”
Gated Self-Matching Networks for Reading Comprehension and Question Answering (2017, background): “Yang et al. (2016) present a fine-grained gating mechanism to dynamically combine word-level and character-level representation and model the interaction between questions and passages.”
ReasoNet: Learning to Stop Reading in Machine Comprehension (2016, neutral): “er/ Table 2: Results on the SQuAD test leaderboard. Single Model Ensemble Model EM F1 EM F1 Logistic Regression Baseline [18] 40.4 51.0 - - Dynamic Chunk Reader [34] 62.5 71.0 - - Fine-Grained Gating [33] 62.5 73.3 - - Match-LSTM [26] 64.7 73.7 67.9 77.0 RaSoR [14] - - 67.4 75.5 Multi-Perspective Ma”
Making Neural QA as Simple as Possible but not Simpler (2017, methodology): “A typical neural architecture consists of an embedding, encoding, interaction-and answer layer (Wang and Jiang, 2017; Yu et al., 2017; Xiong et al., 2017; Seo et al., 2017; Yang et al., 2017; Wang et al., 2017).”
Exploring Question Understanding and Adaptation in Neural-Network-Based Question Answering (2017, neutral): “…Match-LSTM with Ans-Ptr (Boundary) [Wang and Jiang, 2016] 60.474 70.695 Dynamic Chunk Reader [Yu et al., 2016] 62.499 70.956 Fine-Grained Gating [Yang et al., 2016] 62.446 73.327 Match-LSTM with Bi-Ans-Ptr (Boundary) [Wang and Jiang, 2016] 64.744 73.743 Multi-Perspective Matching [Wang et al.,…”
Cold Fusion: Training Seq2Seq Models Together with Language Models (2017, methodology): “Second, we employ fine-grained (FG) gating mechanism [9].”
S-Net: From Answer Extraction to Answer Generation for Machine Reading Comprehension (2017, background): “Other works which only focus on the SQuAD dataset may also be applied on the MS-MARCO dataset (Yu et al., 2016; Lee et al., 2016; Yang et al., 2016).”
Ruminating Reader: Reasoning with Gated Multi-hop Attention (2017, neutral): “ingle model section from April 23, 2017, the time of submission. There are other unpublished systems shown on leaderboard, including Document Reader and r-net. a Rajpurkar et al. (2016);b Yu et al. c Yang et al. (2017 );d Wang and Jiang 2016 e Xiong et al. (2017);f Seo et al. g Lee et al. (2016);h W”
Visual Attention Model for Name Tagging in Multimodal Social Media (2018, methodology): “We propose to use a visual modulation gate, similar to (Miyamoto and Cho, 2016; Yang et al., 2016a), to dynamically control the combination of visual features and word representation generated by BLSTM at word-level, before feed them into the CRF layer for tag prediction.”
Effective Subword Segmentation for Text Comprehension (2018, methodology): “Character-level embedding has been broadly used to refine the word representation [25], [27], [31], [51], showing benefi-”
Can Small and Synthetic Benchmarks Drive Modeling Innovation? A Retrospective Study of Question Answering Modeling Approaches (2021, background): “Researchers mostly focused their efforts on (1) designing better sequence encoders for passages and questions (Lee et al., 2016; Yang et al., 2017; Yu et al., 2018, inter alia ) and (2) proposing improved attention mechanisms for question-passage interactions (Wang and Jiang, 2017; Seo et al.,…”
Subword-augmented Embedding for Cloze Reading Comprehension (2018, background): “Recently, character embeddings are widely used to enrich word representations (Kim et al., 2016; Yang et al., 2017; Luong and Manning, 2016; Huang et al., 2018).”
Structural Embedding of Syntactic Trees for Machine Comprehension (2017, background): “Many machine comprehension models include both pre-trained embeddings and variable embeddings that can be changed through training stage [15, 22].”
Making Neural Machine Reading Comprehension Faster (2019, methodology): “Most of the papers that approach to solve SQuAD problem have utilized RNN based models [3, 5, 7, 9, 13, 14, 16, 17, 20, 22–24, 26, 29–31, 34–39, 41, 43] and they follow a similar chain of processes.”
Effective Character-Augmented Word Embedding for Machine Reading Comprehension (2018, methodology): “Results marked with ‡ are of previously published works [8,7,32].”