LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention

By Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, Yuji Matsumoto. Conference on Empirical Methods in Natural Language Processing, 2020

780 citations recorded by the source corpus.

Entity representations are useful in natural language tasks involving entities. In this paper, we propose new pretrained contextualized representations of words and entities based on the bidirectional transformer. The proposed model treats words and entities in a given text as independent tokens, and outputs contextualized representations of them. Our model is trained using a new pretraining task based on the masked language model of BERT. The task involves predicting randomly masked words and entities in a large entity-annotated corpus retrieved from Wikipedia. We also propose an entity-aware self-attention mechanism that is an extension of the self-attention mechanism of the transformer, and considers the types of tokens (words or entities) when computing attention scores. The proposed model achieves impressive empirical performance on a wide range of entity-related tasks. In particular, it obtains state-of-the-art results on five well-known datasets: Open Entity (entity typing), TACRED (relation classification), CoNLL-2003 (named entity recognition), ReCoRD (cloze-style question answering), and SQuAD 1.1 (extractive question answering). Our source code and pretrained representations are available at this https URL.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/eedf2748a9a1ba2779cde95fd8bad9c2260d5317

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

SpanNER: Named Entity Re-/Recognition as Span Prediction (2021, background): “, 2020), span classification (Jiang et al., 2020; Ouchi et al., 2020; Yamada et al., 2020), and dependency parsing tasks (Yu et al.”
Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph (2023, background): “Require: Input x , LLM π , depth limit D max sample limit N .”
Relational World Knowledge Representation in Contextual Language Models: A Review (2021, background): “, 2020) 3 3 3 3 3 Treating entities as tokens (Yamada et al., 2020; Poerner et al., 2020) 3 3 3 3 3”
Unified Named Entity Recognition as Word-Word Relation Classification (2021, background): “Span-based Approaches There have been several studies that cast NER as span-level classiﬁcation, i.e., enumerating all possible spans, and determining if they are valid mentions and the types (Xu et al. 2017; Luan et al. 2019; Yamada et al. 2020).”
A Unified Generative Framework for Various NER Subtasks (2021, methodology): “We do not compare with Yamada et al. (2020) 2020) are based on span-level classiﬁcations, while our method is based on the entity sequence generation.”
Simple Entity-Centric Questions Challenge Dense Retrievers (2021, background): “, 2020) or LUKE (Yamada et al., 2020) to better recall long-tail entities.”
KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction (2021, methodology): “SPANBERT [30] adopt knowledge to enhance learning objectives, KNOWBERT [38] propose to incorporate knowledge into input features, and LUKE [52] leverage knowledge to improve model architectures.”
The Case for Translation-Invariant Self-Attention in Transformer-Based Language Models (2021, methodology): “Zeng et al. (2020) used relative attention in speech synthesis, letting each query interact with separate matrix transformations for each key vector, depending on their relative-distance offset.”
A large language model for electronic health records (2022, methodology): “1 18 and SQuAD2.”
ExplainaBoard: An Explainable Leaderboard for NLP (2021, background): “We can see that although LUKE surpasses FLERT holistically, it performs worse when dealing with PERSON entities.”
BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining (2022, methodology): “Relation Extraction GLRE [23], REBEL [24], seq2rel [25] KD-DTI [14], BC5CDR [13], DDI [15] Question Answering QA-Net [26], LUKE [27], BERT [2], PubMedBERT [9], BioELECTRA [28], LinkBERT [29] PubMedQA [16], BioASQ [30, 31] Document Classiﬁcation BERT [2], BlueBERT [8], SciBERT [18], SPECTER [32],…”
Few-Shot Named Entity Recognition: A Comprehensive Study (2020, background): “Masked words and entities are jointly trained for prediction in (Yamada et al., 2020) with entity-aware self-attention.”