Deep Semantic Role Labeling with Self-Attention

By Zhixing Tan, Mingxuan Wang, Jun Xie, Yidong Chen, Xiaodong Shi. 2017

320 citations recorded by the source corpus.

Semantic Role Labeling (SRL) is believed to be a crucial step towards natural language understanding and has been widely studied. Recent years, end-to-end SRL with recurrent neural networks (RNN) has gained increasing attention. However, it remains a major challenge for RNNs to handle structural information and long range dependencies. In this paper, we present a simple and effective architecture for SRL which aims to address these problems. Our model is based on self-attention which can directly capture the relationships between two tokens regardless of their distance. Our single model achieves F1=83.4 on the CoNLL-2005 shared task dataset and F1=82.7 on the CoNLL-2012 shared task dataset, which outperforms the previous state-of-the-art results by 1.8 and 1.0 F1 score respectively. Besides, our model is computationally efficient, and the parsing speed is 50K tokens per second on a single Titan X GPU.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Dissecting Contextual Word Embeddings: Architecture and Representation (2018, positive): “In addition to machine translation, it has also provided strong results for Penn Treebank constituency parsing (Kitaev and Klein, 2018) and semantic role labeling (Tan et al., 2018).”
The Natural Language Decathlon: Multitask Learning as Question Answering (2018, background): “State-of-the-art approaches treat SRL as a tagging problem [He et al., 2017], make use of that specific structure to constrain decoding, and mix recurrent and self-attentive layers [Tan et al., 2017].”
Towards Open Intent Discovery for Conversational Text (2019, methodology): “In particular, we use a multi-head self-attention mechanism [20, 30, 44, 46] that jointly attends to information at different positions of the input sequence, with multiple individual attention functions and separately normalized parameters called attention heads.”
Linguistically-Informed Self-Attention for Semantic Role Labeling (2018, methodology): “In these experiments we also compare to Tan et al. (2018), the previous state-of-the art SRL model using gold predicates and standard embeddings.”
Towards Making the Most of BERT in Neural Machine Translation (2019, background): “2013; Pennington, Socher, and Manning 2014) to improve performance on supervised tasks, such as named entity recognition, POS tagging semantic role labelling, classification, and sentiment analysis (Collobert et al. 2011; Socher et al. 2013; Wang and Zheng 2015; Tan et al. 2018).”
Jointly Predicting Predicates and Arguments in Neural Semantic Role Labeling (2018, background): “” Recent high-performing SRL models (He et al., 2017; Marcheggiani et al., 2017; Tan et al., 2018) are BIO-taggers, labeling argument spans for a single predicate at a time (as shown in Figure 1).”
A Survey of the Usages of Deep Learning for Natural Language Processing (2020, methodology): “[123] used a self-attentional model for semantic role labeling and a subtask of semantic parsing, achieving excellent results.”
Structured Dialogue Policy with Graph Neural Networks (2018, unknown): “It is a variant of dropout and adopted in self-attention models (Tan et al., 2018) and graph convolutional networks (Kipf and Welling, 2017).”
Strategies for Structuring Story Generation (2019, methodology): “Conditioned upon the prompt, we generate an SRL decomposition of the story by concatenating the predicates and arguments identified by a pretrained model (He et al., 2017; Tan et al., 2018)1 and separating sentences with delimiter tokens.”
Joint Slot Filling and Intent Detection via Capsule Neural Networks (2018, background): “Recently, Shen et al. (2017); Tan et al. (2017) introduce the self-attention mechanism for CRF-free sequential labeling.”
DySAT: Deep Neural Representation Learning on Dynamic Graphs via Self-Attention Networks (2020, background): “Recent advances in many NLP tasks have demonstrated the superiority of self-attention both in efficiency and performance [28, 29, 33].”
Natural Language Processing Advancements By Deep Learning: A Survey (2020, methodology): “Current state-of-the-art methods employ joint prediction of predicates and arguments [94], novel word representation approaches [95], and self-attention models [96]; see Table IV.”
A Stack-Propagation Framework with Token-Level Intent Detection for Spoken Language Understanding (2019, methodology): “Self-attention is a very effective method of leveraging context-aware features over variable-length sequences for natural language processing tasks (Tan et al., 2018; Zhong et al., 2018).”
Fast Directional Self-Attention Mechanism (2018, methodology): “[35] designed a deep attentive neural net by stacking multi-head self-attention, named as deepatt, to perform context fusion, whose output is then passed to a neural classifier to make the final decision.”