Enhancing Pre-Trained Language Representations with Rich Knowledge for Machine Reading Comprehension

By An Yang, Quan Wang, Jing Liu, Kai Liu, Yajuan Lyu, Hua Wu, Qiaoqiao She, Sujian Li. 2019

152 citations recorded by the source corpus.

Machine reading comprehension (MRC) is a crucial and challenging task in NLP. Recently, pre-trained language models (LMs), especially BERT, have achieved remarkable success, presenting new state-of-the-art results in MRC. In this work, we investigate the potential of leveraging external knowledge bases (KBs) to further improve BERT for MRC. We introduce KT-NET, which employs an attention mechanism to adaptively select desired knowledge from KBs, and then fuses selected knowledge with BERT to enable context- and knowledge-aware predictions. We believe this would combine the merits of both deep LMs and curated KBs towards better MRC. Experimental results indicate that KT-NET offers significant and consistent improvements over BERT, outperforming competitive baselines on ReCoRD and SQuAD1.1 benchmarks. Notably, it ranks the 1st place on the ReCoRD leaderboard, and is also the best single model on the SQuAD1.1 leaderboard at the time of submission (March 4th, 2019).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Retrieval-Augmented Generation, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Percy Liang
  2. Mohit Bansal
  3. Yejin Choi
  4. Xiang Ren
  5. Noah A. Smith
  6. Luke S. Zettlemoyer
  7. Hannaneh Hajishirzi
  8. Jianfeng Gao

In Computer Vision and Pattern Recognition

  1. Chitta Baral
  2. Percy Liang
  3. Xiang Ren
  4. Hannaneh Hajishirzi
  5. Yejin Choi
  6. Noah A. Smith
  7. Mohit Bansal
  8. Sameer Singh

In Retrieval-Augmented Generation

  1. Xiang Ren
  2. Noah A. Smith
  3. Christopher D. Manning
  4. Percy Liang
  5. Xuanjing Huang
  6. Peter Clark
  7. Bill Yuchen Lin

In Natural Language Processing

  1. Yejin Choi
  2. Percy Liang
  3. Mohit Bansal
  4. Noah A. Smith
  5. Hannaneh Hajishirzi
  6. Luke S. Zettlemoyer
  7. Chitta Baral
  8. Xiang Ren

Citations in the corpus