Simple BERT Models for Relation Extraction and Semantic Role Labeling

By Peng Shi, Jimmy J. Lin. 2019

479 citations recorded by the source corpus.

We present simple BERT-based models for relation extraction and semantic role labeling. In recent years, state-of-the-art performance has been achieved using neural models by incorporating lexical and syntactic features such as part-of-speech tags and dependency trees. In this paper, extensive experiments on datasets for these two tasks show that without using any external features, a simple BERT-based model can achieve state-of-the-art performance. To our knowledge, we are the first to successfully apply BERT in this manner. Our models provide strong baselines for future research.

Subfields: Artificial Intelligence, Molecular Biology, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Natural Language Processing

Citations in the corpus

Reasoning Over Semantic-Level Graph for Fact Checking (2019, methodology): “• For each sentence, we parse it to tuples 2 with an off-the-shelf SRL toolkit developed by Al-lenNLP 3 , which is a re-implementation of a BERT-based model (Shi and Lin, 2019).”
Tailor: Generating and Perturbing Text with Semantic Controls (2021, methodology): “Our experiments use the opensourced implementation of Shi and Lin (2019): demo. allennlp.org/semantic-role-labeling , with a test F1 of 86.5 on the Ontonotes 5.0 dataset (Pradhan et al., 2013).”
Learning from Task Descriptions (2020, methodology): “Zero-shot learning has been widely adopted including for classi-ﬁcation (Dauphin et al., 2013), entity typing (Ma et al., 2016; Zhou et al., 2018) and relation extraction (Levy et al., 2017; Shi and Lin, 2019).”
Temporal Common Sense Acquisition with Minimal Supervision (2020, methodology): “We ﬁrst process the entire Gigaword (Graff et al., 2003) corpus and use AllenNLP’s SRL model (Gardner et al., 2018; Shi and Lin, 2019) to annotate all sentences.”
Fine-Grained Video-Text Retrieval With Hierarchical Graph Reasoning (2020, methodology): “Then we employ an off-the-shelf semantic role parsing toolkit [30] to obtain verbs, noun phrases in C as well as the semantic role of each noun phrase to the corresponding verb (details of semantic roles are given in the supplementary).”
TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task (2020, unknown): “semi-supervised pre-training: fine-tuning of language representations pre-trained on token-level (Alt et al., 2019; Shi and Lin, 2019) or span-level (Joshi et al.”
Document-Level Relation Extraction with Adaptive Thresholding and Localized Context Pooling (2020, methodology): “…et al., 2017), graph-based methods (Miwa and Bansal, 2016; Zhang et al., 2018; Guo et al., 2019; Wu et al., 2019a), transformer-based methods (Alt et al., 2019; Shi and Lin, 2019), and pre-training methods (Zhang et al., 2019; Soares et al., 2019) have been shown effective in tackling this problem.”
DeepStruct: Pretraining of Language Models for Structure Prediction (2022, methodology): “…models: (i) Dep and Span (Li et al., 2019): this model formulates semantic role labeling as an end-to-end dependency parsing task, (ii) BERT SRL (Shi and Lin, 2019): it is a sequence-tagging version of BERT, (iii) TANL (Paolini et al., 2021) Evaluation Details Sentences with multiple target…”
Structured Prediction as Translation between Augmented Natural Languages (2021, result): “We compare our results with Dependency and Span SRL (Li et al., 2019d), which uses a Bi-LSTM with highway connection and biafﬁne scorers, and BERT-SRL (Shi & Lin, 2019), BERT-based model which predicts the spans based on the contextual and positional embeddings.”
Document-Level Event Argument Extraction by Conditional Generation (2021, unknown): “Since the baseline BERT-CRF model (Shi and Lin, 2019) cannot handle new labels directly, we exclude it from comparison.”
Large language models are few-shot clinical information extractors (2022, methodology): “For end-to-end relation extraction, we first used the token-level baseline to extract entity spans, then used the technique from Shi and Lin (2019) to classify whether each pair of entities was related.”
NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation (2021, methodology): “This perturbation adds noise to all types of English text sources (sentence, paragraph, etc.) proportional to the number of arguments identified by SRL BERT (Shi and Lin, 2019b).”
DARE: Data Augmented Relation Extraction with GPT-2 (2020, background): “…the latest advances in language modelling with Transformer-based architectures ( Radford et al., 2018a; Devlin et al., 2018; Radford et al., 2018b) have been quickly employed to also deal with RE tasks (Soares et al., 2019; Lin et al., 2019; Shi and Lin, 2019; Papanikolaou et al., 2019).”
Improving Robustness by Augmenting Training Sentences with Predicate-Argument Structures (2020, methodology): “We use the PropBank-style semantic role labeling model of Shi and Lin (2019), which has state-of-the-art results on the CoNLL-2009 dataset.”
Paired Representation Learning for Event and Entity Coreference (2020, methodology): “Preprocessing: Argument generation We use AI2 SRL sys-tem 3 , which is a reimplementation of Shi and Lin (2019), to predict arguments for each event mention, and map to the annotation of gold arguments.”
Do Opt-Outs Really Opt Me Out? (2022, methodology): “The semantic arguments are analyzed by using a semantic role labeling model (SRL) [6] based on Roberta-base contextualized word embeddings and trained on the CoNLL2012 (OntoNotes 5) largescale natural language dataset [92].”