Improving Relation Extraction by Pre-trained Language Representations

By Christoph Alt, Marc Hübner, Leonhard Hennig. 2019

94 citations recorded by the source corpus.

Current state-of-the-art relation extraction methods typically rely on a set of lexical, syntactic, and semantic features, explicitly computed in a pre-processing step. Training feature extraction models requires additional annotated language resources, which severely restricts the applicability and portability of relation extraction to novel languages. Similarly, pre-processing introduces an additional source of error. To address these limitations, we introduce TRE, a Transformer for Relation Extraction, extending the OpenAI Generative Pre-trained Transformer [Radford et al., 2018]. Unlike previous relation extraction models, TRE uses pre-trained deep language representations instead of explicit linguistic features to inform the relation classification and combines it with the self-attentive Transformer architecture to effectively model long-range dependencies between entity mentions. TRE allows us to learn implicit linguistic features solely from plain text corpora by unsupervised pre-training, before fine-tuning the learned language representations on the relation extraction task. TRE obtains a new state-of-the-art result on the TACRED and SemEval 2010 Task 8 datasets, achieving a test F1 of 67.4 and 87.1, respectively. Furthermore, we observe a significant increase in sample efficiency. With only 20% of the training examples, TRE matches the performance of our baselines and our model trained from scratch on 100% of the TACRED dataset. We open-source our trained models, experiments, and source code.

Subfields: Natural Language Processing, Information Extraction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Information Extraction

Citations in the corpus

Knowledge Enhanced Contextual Word Representations (2019, neutral): “4 Alt et al. (2019) GPT 70.”
Probing Linguistic Features of Sentence-Level Representations in Relation Extraction (2020, background): “…state-of-the-art performance on standard datasets like SemEval2010 Task 8 (dos Santos et al., 2015; Wang et al., 2016; Lee et al., 2019), TA-CRED (Zhang et al., 2018; Alt et al., 2019b; Peters et al., 2019; Joshi et al., 2019), and NYT (Lin et al., 2016; Vashishth et al., 2018; Alt et al., 2019a).”
Document-Level Relation Extraction with Adaptive Thresholding and Localized Context Pooling (2020, methodology): “…Zhang et al., 2017), graph-based methods (Miwa and Bansal, 2016; Zhang et al., 2018; Guo et al., 2019; Wu et al., 2019a), transformer-based methods (Alt et al., 2019; Shi and Lin, 2019), and pre-training methods (Zhang et al., 2019; Soares et al., 2019) have been shown effective in tackling this…”
Simple BERT Models for Relation Extraction and Semantic Role Labeling (2019, background): “Alt et al. (2019) leverage the pretrained language model GPT (Radford et al., 2018) and achieves better recall than our system.”
TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task (2020, unknown): “Many current approaches use unsupervised or semi-supervised pre-training: ﬁne-tuning of language representations pre-trained on token-level (Alt et al., 2019; Shi and Lin, 2019) or span-level (Joshi et al., 2019), ﬁne-tuning of knowledge enhanced word representations that are pre-trained on…”