BERT is Not a Knowledge Base (Yet): Factual Knowledge vs. Name-Based Reasoning in Unsupervised QA

By Nina Poerner, Ulli Waltinger, Hinrich Schütze. 2019

80 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

Citations in the corpus

Entities as Experts: Sparse Memory Access with Entity Supervision (2020, neutral): “…models (Ahn et al., 2016; Yang et al., 2016; Logan et al., 2019; Zhang et al., 2019; Levine et al., 2019; Xiong et al., 2019a; Peters et al., 2019; Poerner et al., 2019; Wang et al., 2020 Memory Augmented Neural Networks Our entity memory layer is closely tied to memory-based neural layers…”
How Can We Know What Language Models Know? (2019, methodology): “Similar extensions have been applied to pre-trained LMs like BERT, where contextualized representations are enhanced with entity embeddings (Zhang et al., 2019; Peters et al., 2019; Pörner et al., 2019).”
Factual Probing Is [MASK]: Learning vs. Learning to Recall (2021, methodology): “UHN stands for UnHelpfulNames (Poerner et al., 2019), which is a subset of LAMA where questions with helpful entity names were deleted.”
A Primer in BERTology: What We Know About How BERT Works (2020, background): “…proposed classification of attention head types: • attending to the word itself, to previous/next words and to the end of the sentence (Raganato and Tiedemann, 2018); • attending to previous/next tokens, [CLS], [SEP], punctuation, and “attending broadly” over the sequence (Clark et al., 2019); •…”
Information-Theoretic Probing with Minimum Description Length (2020, background): “…errors (Linzen et al., 2016; Gulordava et al., 2018; Tran et al., 2018; Marvin and Linzen, 2018) or understanding what language models know when applying them as knowledge bases or in question answering settings (Radford et al., 2019; Petroni et al., 2019; Poerner et al., 2019; Jiang et al., 2019).”
Pre-trained models for natural language processing: A survey (2020, neutral): “Despite the surprising findings of LAMA, it has also been questioned by subsequent work [141, 82].”
Knowledge-Augmented Methods for Natural Language Processing (2023, background): “As a result, these models struggle to grasp external world knowledge about concepts, relations, and common sense [32, 38].”
Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoning (2020, background): “More recently, analysis of relational knowledge contained in pre-trained BERT models has been the subject of different studies (Petroni et al., 2019; Poerner et al., 2019).”
An Information Theoretic View on Selecting Linguistic Probes (2020, background): “Such tasks include probing syntax (Hewitt and Manning, 2019; Lin et al., 2019; Tenney et al., 2019a), semantics (Yaghoobzadeh et al., 2019), discourse features (Chen et al., 2019; Liu et al., 2019; Tenney et al., 2019b), and commonsense knowledge (Petroni et al., 2019; Poerner et al., 2019).”
Can Generative Pre-trained Language Models Serve As Knowledge Bases for Closed-book QA? (2021, background): “However, Poerner et al. (2019) argue that BERT uses some superﬁcial cues such as stereotypical characters to solve factual questions.”
CoLAKE: Contextualized Language and Knowledge Embedding (2020, methodology): “Subsequently, a more ”factual” subset of LAMA, LAMA-UHN (P¨orner et al., 2019), is constructed by ﬁltering out easy-to-answer samples.”