Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model

By Wenhan Xiong, Jingfei Du, William Yang Wang, Veselin Stoyanov. 2019

216 citations recorded by the source corpus.

Recent breakthroughs of pretrained language models have shown the effectiveness of self-supervised learning for a wide range of natural language processing (NLP) tasks. In addition to standard syntactic and semantic NLP tasks, pretrained models achieve strong improvements on tasks that involve real-world knowledge, suggesting that large-scale language modeling could be an implicit method to capture knowledge. In this work, we further investigate the extent to which pretrained models such as BERT capture knowledge using a zero-shot fact completion task. Moreover, we propose a simple yet effective weakly supervised pretraining objective, which explicitly forces the model to incorporate knowledge about real-world entities. Models trained with our new objective yield significant improvements on the fact completion task. When applied to downstream tasks, our model consistently outperforms BERT on four entity-related question answering datasets (i.e., WebQuestions, TriviaQA, SearchQA and Quasar-T) with an average 2.7 F1 improvements and a standard fine-grained entity typing dataset (i.e., FIGER) with 5.7 accuracy gains.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Entities as Experts: Sparse Memory Access with Entity Supervision (2020, background): “…seeking to inject knowledge into language models (Ahn et al., 2016; Yang et al., 2016; Logan et al., 2019; Zhang et al., 2019; Levine et al., 2019; Xiong et al., 2019a; Peters et al., 2019; Poerner et al., 2019; Wang et al., 2020 Memory Augmented Neural Networks Our entity memory layer is closely…”
Progressively Pretrained Dense Corpus Index for Open-Domain Question Answering (2020, background): “Speciﬁcally, improvements include stronger reading comprehension models (Wang et al., 2018; Yang et al., 2019; Xiong et al., 2019; Min et al., 2019a) and paragraph reranking models (Wang et al., 2017; Lin et al., 2018) that assign more accurate relevance scores to the retrieved paragraphs.”
KILT: a Benchmark for Knowledge Intensive Language Tasks (2020, background): “, 2013; Surdeanu and Ji, 2014) seems outof-fashion, recent results show that they remain contenders (Fan et al., 2019a; Xiong et al., 2019).”
Symbolic Knowledge Distillation: from General Language Models to Commonsense Models (2021, background): “This makes Y the set of all strings, over which LMs deﬁne probability.”
Pre-Trained Models: Past, Present and Future (2021, background): “, 2021) or their alignments with the text (Xiong et al., 2019; Sun et al., 2019b).”
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters (2020, methodology): “WKLM + Ranking (Xiong et al., 2020) is a WKLM paragraph reader plus with a BERT based paragraph ranker with distant-supervised data to assign each paragraph a relevance score.”
Pre-trained models for natural language processing: A survey (2020, methodology): “D CBOW-NS [129], ELECTRA [24] NSP BERT [36], UniLM [39] SOP ALBERT [93], StructBERT [193] Extensions Knowledge-Enriched ERNIE(THU) [214], KnowBERT [136], K-BERT [111] SentiLR [83], KEPLER [195], WKLM [202] Multilingual XLU mBERT [36], Unicoder [68], XLM [27], XLM-R [28], MultiFit [42] XLG MASS [160]”
KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation (2019, background): “Besides, Logan et al. (2019); Hayashi et al. (2020) utilize relations between entities inside one sentence to help train better generation models, and Xiong et al. (2019) adopt entity replacement knowledge learning for improving entity-related tasks.”
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models (2021, methodology): “Variants of TyQ have been used before (Xiong et al., 2020).”
ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning (2020, background): “Some enforce PLMs to memorize information about real-world entities and propose novel pretraining objectives (Xiong et al., 2019; Wang et al., 2019; Sun et al., 2020; Yamada et al., 2020).”
KB-VLP: Knowledge Based Vision and Language Pretraining (2021, background): “…using knowledge during pretraining of language models (Yu et al., 2020; Xu et al., 2021; Ros-set et al., 2021; Zhou et al., 2020; He et al., 2020a; Xiong et al., 2019; He et al., 2020b; Agarwal et al., 2021), most have not been applied to multi-modal transformers (e.g. for vision and language).”
Measuring Causal Effects of Data Statistics on Language Model's 'Factual' Predictions (2022, background): “Following Xiong et al. (2020) Table 3: A subset of the table for computing the causal effect for the subject-object co-occurrences hypothesis.”
MATE: Multi-view Attention for Table Transformer Efficiency (2021, background): “The observed errors suggest potential gains by improving the entity (Xiong et al., 2020) and numerical (Andor et al., 2019) reasoning skills.”
A Survey of Knowledge-enhanced Text Generation (2020, background): “ pre-training and knowledge enhancement. Existing work in NLP community has explored the idea of explicitly modeling task-specific knowledge by designing pre-training tasks on massive unlabelled data [55, 159, 178]. It is thus important to design novel ACM Comput. Surv., Vol. 1, No. 1, Article 1. Pu”
Deep Bidirectional Language-Knowledge Graph Pretraining (2022, unknown): “How to effectively combine text and KGs for pretraining is an open problem and presents challenges.”
Pre-training Text-to-Text Transformers for Concept-centric Common Sense (2020, methodology): “Towards augmenting PTLMs with more knowledge, prior works mainly focus on training larger models (Brown et al., 2020), adding speciﬁc architectures to exploit external knowledge (Peters et al., 2019), or incorporating knowledge bases for pre-training (Xiong et al., 2020).”