Enhancing Pre-Trained Language Representations with Rich Knowledge for Machine Reading Comprehension
By An Yang, Quan Wang, Jing Liu, Kai Liu, Yajuan Lyu, Hua Wu, Qiaoqiao She, Sujian Li. 2019
152 citations recorded by the source corpus.
Machine reading comprehension (MRC) is a crucial and challenging task in NLP. Recently, pre-trained language models (LMs), especially BERT, have achieved remarkable success, presenting new state-of-the-art results in MRC. In this work, we investigate the potential of leveraging external knowledge bases (KBs) to further improve BERT for MRC. We introduce KT-NET, which employs an attention mechanism to adaptively select desired knowledge from KBs, and then fuses selected knowledge with BERT to enable context- and knowledge-aware predictions. We believe this would combine the merits of both deep LMs and curated KBs towards better MRC. Experimental results indicate that KT-NET offers significant and consistent improvements over BERT, outperforming competitive baselines on ReCoRD and SQuAD1.1 benchmarks. Notably, it ranks the 1st place on the ReCoRD leaderboard, and is also the best single model on the SQuAD1.1 leaderboard at the time of submission (March 4th, 2019).
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Retrieval-Augmented Generation, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Percy Liang
- Mohit Bansal
- Yejin Choi
- Xiang Ren
- Noah A. Smith
- Luke S. Zettlemoyer
- Hannaneh Hajishirzi
- Jianfeng Gao
In Computer Vision and Pattern Recognition
- Chitta Baral
- Percy Liang
- Xiang Ren
- Hannaneh Hajishirzi
- Yejin Choi
- Noah A. Smith
- Mohit Bansal
- Sameer Singh
In Retrieval-Augmented Generation
- Xiang Ren
- Noah A. Smith
- Christopher D. Manning
- Percy Liang
- Xuanjing Huang
- Peter Clark
- Bill Yuchen Lin
In Natural Language Processing
- Yejin Choi
- Percy Liang
- Mohit Bansal
- Noah A. Smith
- Hannaneh Hajishirzi
- Luke S. Zettlemoyer
- Chitta Baral
- Xiang Ren
Citations in the corpus
- StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding (2019, neutral): “Table 2: Accuracy (%) on the SNLI dataset. 5 System Dev set Test set EM F1 EM F1 Human - - 82.3 91.2 XLNet(single+DA) [32] 88.9 94.5 89.9 85.0 BERT(ensemble+DA) [6] 86.2 92.2 87.4 93.2 KT-NET(single) [31] 85.1 81.7 85.9 92.4 BERT(single+DA) [6] 84.2 91.1 85.1 91.8 QANet(ensemble+DA) [33] - - 84.5 90”
- Knowledge Enhanced Contextual Word Representations (2019, background): “Task-specific KB architectures Other work has focused on integrating KBs into neural architectures for specific downstream tasks (Yang and Mitchell, 2017; Sun et al., 2018; Chen et al., 2018; Bauer et al., 2018; Mihaylov and Frank, 2018; Wang and Jiang, 2019; Yang et al., 2019).”
- Pre-trained models for natural language processing: A survey (2020, background): “ows injecting factual knowledge during fine-tuning on downstream tasks. Guan et al. [52] employed commonsense knowledge bases, ConceptNet and ATOMIC, to enhance GPT-2 for story generation. Yang et al. [207] proposed a knowledge-text fusion model to acquire related linguistic and factual knowledge for”
- Knowledge Fusion and Semantic Knowledge Ranking for Open Domain Question Answering (2020, background): “Other models such as in the work of (Mi-haylov and Frank, 2018; Chen et al., 2018; Yang et al., 2019; Wang and Jiang, 2019) embed syntactic or semantic knowledge into knowledge enriched embeddings.”
- Deep Bidirectional Language-Knowledge Graph Pretraining (2022, unknown): “…performance improvements on a broader set of NLP tasks including QA, reasoning and text classification. answering (Lin et al. 2019; Feng et al. 2020; Lv et al. 2020; Wang et al. 2022; Mihaylov & Frank 2018; Yang et al. 2019; Sun et al. 2018, 2019a; Yan et al. 2021; Sun et al. 2022; Xu et al. 2022).”
- QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering (2021, background): “To provide more explicit and interpretable knowledge, several works integrate structured knowledge (KGs) into LMs (Mihaylov and Frank, 2018; Lin et al., 2019; Wang et al., 2019a; Yang et al., 2019; Wang et al., 2020b; Bosselut et al., 2021).”
- GenericsKB: A Knowledge Base of Generic Statements (2020, background): “, (Icarte et al., 2017; Wang et al., 2018; Yang et al., 2019; Peters et al., 2019; Liu et al., 2019; Paul and Frank, 2019).”
- How Additional Knowledge can Improve Natural Language Commonsense Question Answering (2019, background): “External Knowledge: Models that integrate external knowledge have been introduced in (Mihaylov and Frank, 2018; Chen et al., 2018; Yang et al., 2019; Wang and Jiang, 2019).”
- GreaseLM: Graph REASoning Enhanced Language Models for Question Answering (2022, methodology): “Other works seek to use one modality to ground the other, such as using an encoded representation of a linked KG to augment the textual representation of a QA example ( e.g. , Knowledgeable Reader, Mihaylov & Frank, 2018; KagNet, Lin et al., 2019; KT-NET, Yang et al., 2019).”
- Domain-specific Knowledge Graphs: A survey (2020, background): “In fact, various of these massive publicly-available data islands have been harvested from the Web as being a key source of knowledge to benefit numerous Artificial Intelligence and smart systems [12], such as recommender systems [13], decision support systems (DSSs) [14], and intelligent QA systems”
- Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification (2021, unknown): “, 2020) and the fine-tuning stage (Yang et al., 2019; Guan et al., 2020).”
- Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question Answering (2020, background): “C L ] 1 M ay 2 02 0 this kind of questions (Mihaylov and Frank, 2018; Lin et al., 2019; Wang et al., 2019; Yang et al., 2019).”