End-to-end learning of semantic role labeling using recurrent neural networks

By Jie Zhou, Wei Xu. 2015

360 citations recorded by the source corpus.

Semantic role labeling (SRL) is one of the basic natural language processing (NLP) problems. To this date, most of the successful SRL systems were built on top of some form of parsing results (Koomen et al., 2005; Palmer et al., 2010; Pradhan et al., 2013), where pre-defined feature templates over the syntactic structure are used. The attempts of building an end-to-end SRL learning system without using parsing were less successful (Collobert et al., 2011). In this work, we propose to use deep bi-directional recurrent network as an end-to-end system for SRL. We take only original text information as input feature, without using any syntactic knowledge. The proposed algorithm for semantic role labeling was mainly evaluated on CoNLL-2005 shared task and achieved F1 score of 81.07. This result outperforms the previous state-of-the-art system from the combination of different parsing trees or models. We also obtained the same conclusion with F1 = 81.27 on CoNLL2012 shared task. As a result of simplicity, our model is also computationally efficient that the parsing speed is 6.7k tokens per second. Our analysis shows that our model is better at handling longer sentences than traditional models. And the latent variables of our model implicitly capture the syntactic structure of a sentence.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation (2016, background): “For such a deep and large network, it is not easy to determine the tuning strategy and this would be considered in future work.”
Deep Contextualized Word Representations (2018, methodology): “He et al. (2017) modeled SRL as a BIO tagging problem and used an 8-layer deep biLSTM with forward and backward directions interleaved, following Zhou and Xu (2015).”
context2vec: Learning Generic Context Embedding with Bidirectional LSTM (2016, methodology): “More recently, bidirectional recurrent neural networks, and speciﬁcally bidirectional LSTMs, were used in such tasks to learn internal representations of wider sentential contexts (Zhou and Xu, 2015; Lample et al., 2016).”
The Natural Language Decathlon: Multitask Learning as Question Answering (2018, methodology): “(2008), but recent methods have demonstrated that it is not necessary to use syntactic information as additional input (Zhou and Xu, 2015; Marcheggiani et al., 2017).”
AllenNLP: A Deep Semantic Natural Language Processing Platform (2018, background): “…state-of-the-art for a wide range of tasks such as text classiﬁ-cation (Howard and Ruder, 2018), machine translation (Vaswani et al., 2017), semantic role labeling (Zhou and Xu, 2015; He et al., 2017), coreference resolution (Lee et al., 2017a), and semantic parsing (Krishnamurthy et al., 2017).”
Recent Trends in Deep Learning Based Natural Language Processing (2017, methodology): “(Zhou and Xu, 2015) proposed to use bidirectional LSTM to model arbitrarily long context, which proved to be successful without any parsing tree information.”
Natural Language Processing for EHR-Based Computational Phenotyping (2018, background): “tion [146], semantic analysis [147], [148], information retrieval [149], [150], entity recognition [151], [152], relation extraction [153], [154], [155], [156], and event detection [157], [158].”
Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling (2017, background): “, 2008; Johansson and Nugues, 2008), the last generation of SRL models put syntax aside in favor of neural sequence models, namely LSTMs (Zhou and Xu, 2015; Marcheggiani et al., 2017), and outperformed syntactically-driven methods on standard benchmarks.”
Globally Normalized Transition-Based Neural Networks (2016, methodology): “Our work is closest to the work of Weiss et al. (2015), Zhou et al. (2015) and Watanabe and Sumita (2015); in these approaches global normalization is added to the local model of Chen and Manning (2014).”
Deep Semantic Role Labeling: What Works and What’s Next (2017, methodology): “Most recently, several successful LSTM-based architectures have achieved state-of-the-art results in English span-based SRL (Zhou and Xu, 2015), Chinese SRL (Wang et al.”
Dataset and Neural Recurrent Sequence Labeling Model for Open-Domain Factoid Question Answering (2016, methodology): “Following (Huang et al., 2015; Zhou and Xu, 2015), we use CRF on top of evidence LSTMs for sequence labeling.”
Linguistically-Informed Self-Attention for Semantic Role Labeling (2018, background): “…for SRL systems (Levin, 1993; Pun-yakanok et al., 2008), recently deep neural network architectures have surpassed syntactically-informed models (Zhou and Xu, 2015; Marcheg-giani et al., 2017; He et al., 2017; Tan et al., 2018; He et al., 2018), achieving state-of-the art SRL performance with…”
Leveraging Sentence-level Information with Encoder LSTM for Semantic Slot Filling (2016, methodology): “Bi-directional RNN/LSTM have been proposed to capture sentence-level informa-O tion (Mesnil et al., 2015; Zhou and Xu, 2015; Vu et al., 2016).”
Analyzing Linguistic Knowledge in Sequential Model of Sentence (2016, methodology): “With the application of popular deep learning methods, researchers have found that recurrent neural network can successfully model the non-sequential linguistic properties with sequential data input (Vinyals et al., 2015; Zhou and Xu, 2015; Rockt¨aschel et al., 2015).”
A Sequence-to-Sequence Model for Semantic Role Labeling (2018, methodology): “Following Zhou and Xu (2015), if a sentence has np predicates we process the sentence np times, each one with its corresponding predicate-argument structure.”
Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretraining (2020, background): “[10], Intent Classiﬁcation (IC) and Slot Labeling (SL) are jointly predicted on top of BERT, discarding the need of a Conditional Random Fields (CRF) [11].”
Large-Scale QA-SRL Parsing (2018, methodology): “All models are built on an LSTM encoding of the sentence.”
Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Loss (2016, methodology): “…al., 2015), POS tagging (Ling et al., 2015; Wang et al., 2015), transition-based dependency parsing (Ballesteros et al., 2015; Kiperwasser and Goldberg, 2016), ﬁne-grained sentiment analysis (Liu et al., 2015), syntactic chunking (Huang et al., 2015), and semantic role labeling (Zhou and Xu, 2015).”
Natural Language Processing Advancements By Deep Learning: A Survey (2020, neutral): “disciplines utilizing deep learning methods, deep neural networks have been applied to various NLP tasks, including partof-speech tagging [10]–[12], named entity recognition [13], [13], [14], and semantic role labeling [15]–[17].”