Linguistically-Informed Self-Attention for Semantic Role Labeling

By Emma Strubell, Pat Verga, D. Andor, David Weiss, Andrew McCallum. 2018

401 citations recorded by the source corpus.

Current state-of-the-art semantic role labeling (SRL) uses a deep neural network with no explicit linguistic features. However, prior work has shown that gold syntax trees can dramatically improve SRL decoding, suggesting the possibility of increased accuracy from explicit modeling of syntax. In this work, we present linguistically-informed self-attention (LISA): a neural network model that combines multi-head self-attention with multi-task learning across dependency parsing, part-of-speech tagging, predicate detection and SRL. Unlike previous models which require significant pre-processing to prepare linguistic features, LISA can incorporate syntax using merely raw tokens as input, encoding the sequence only once to simultaneously perform parsing, predicate detection and role labeling for all predicates. Syntax is incorporated by training one attention head to attend to syntactic parents for each token. Moreover, if a high-quality syntactic parse is already available, it can be beneficially injected at test time without re-training our SRL model. In experiments on CoNLL-2005 SRL, LISA achieves new state-of-the-art performance for a model using predicted predicates and standard word embeddings, attaining 2.5 F1 absolute higher than the previous state-of-the-art on newswire and more than 3.5 F1 on out-of-domain data, nearly 10% reduction in error. On ConLL-2012 English SRL we also show an improvement of more than 2.5 F1. LISA also out-performs the state-of-the-art with contextually-encoded (ELMo) word representations, by nearly 1.0 F1 on news and more than 2.0 F1 on out-of-domain text.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Energy and Policy Considerations for Modern Deep Learning Research (2020, methodology): “To quantify the computational requirements of developing a new model, in this case study we analyze the logs of all training required to develop Linguistically-Informed Self-Attention (Strubell et al. 2018), a multi-task model that performs four related natural language tasks.”
Multimodal Transformer for Unaligned Multimodal Language Sequences (2019, methodology): “…have also been successfully applied to other tasks, including language modeling (Dai et al., 2018; Baevski and Auli, 2019), semantic role labeling (Strubell et al., 2018), word sense disambiguation (Tang et al., 2018), learning sentence representations (Devlin et al., 2018), and video activity…”
Energy and Policy Considerations for Deep Learning in NLP (2019, methodology): “To quantify the computational requirements of R&D for a new model we study the logs of all training required to develop LinguisticallyInformed Self-Attention (Strubell et al., 2018), a multi-task model that performs part-of-speech tagging, labeled dependency parsing, predicate detection and semantic”
What Does BERT Look at? An Analysis of BERT’s Attention (2019, background): “Several recent works have proposed incorporating syntactic information to improve attention (Eriguchi et al., 2016; Chen et al., 2018; Strubell et al., 2018).”
Are Sixteen Heads Really Better than One? (2019, background): “…was ﬁrst introduced by Vaswani et al. (2017) for NMT and English constituency parsing, and later adopted for transfer learning (Radford et al., 2018; Devlin et al., 2018), language modeling (Dai et al., 2019; Radford et al., 2019), or semantic role labeling (Strubell et al., 2018), among others.”
What do you learn from context? Probing for sentence structure in contextualized word representations (2019, background): “Recent work has consistently demonstrated the strong empirical performance of contextualized word representations, including CoVe (McCann et al., 2017), ULMFit (Howard & Ruder, 2018), ELMo (Peters et al., 2018a; Lee et al., 2018; Strubell et al., 2018; Kitaev & Klein, 2018).”
Emergent linguistic structure in artificial neural networks trained by self-supervision (2020, background): “Explicitly incorporating § Intuitively, the precision score in this case measures what percent of the time the most-attended-to word of a verb is that verb’s direct object. syntactic information to improve attention has been an active area of NLP research (39–41).”
HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization (2019, methodology): “We choose the Transformer because it outperforms CNN and RNN in machine translation (Vaswani et al., 2017), semantic role labeling (Strubell et al., 2018) and other NLP tasks (Devlin et al., 2018).”
Semantically Conditioned Dialog Response Generation via Hierarchical Disentangled Self-Attention (2019, methodology): “Our proposed HDSA is inspired by the linguistically-inform self-attention (Strubell et al., 2018), which combines multi-head self-attention with multi-task NLP tasks to enhance the linguistic awareness of the model.”
KERMIT: Complementing Transformer Architectures with Encoders of Explicit Syntactic Interpretations (2020, background): “Moreover, Munkhdalai and Yu (2017) have specialized LSTM for binary and n-ry trees with their Neural Tree Indexers and Strubell et al. (2018) have encoded syntactic information by using multi-head attention within a transformer architecture.”
Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation (2020, neutral): “, 2018) and other downstream tasks (Strubell et al., 2018; Devlin et al., 2019; Bosselut et al., 2019).”
Natural Language Processing Advancements By Deep Learning: A Survey (2020, background): “leverage both multi-task learning and self-attention for effective utilization of the syntactic information for SRL [93].”
Few-shot Text Classification with Distributional Signatures (2019, background): “…from the source to the target by either ﬁne-tuning a pre-trained encoder (Howard & Ruder, 2018; Peters et al., 2018; Radford et al., 2018; Bertinetto et al., 2019), or multi-task learning with a shared encoder (Collobert & Weston, 2008; Liu et al., 2015; Luong et al., 2015; Strubell et al., 2018).”
Towards Understanding Neural Machine Translation with Word Importance (2019, background): “Further analyses show that important words are of distinct syntactic categories on different language pairs, which might support the viewpoint that essential inductive bias should be introduced into the model design (Strubell et al., 2018).”
Modeling Recurrence for Transformer (2019, background): “A number of recent efforts have explored ways to improve multi-head SAN by encouraging individual attention heads to extract distinct information (Strubell et al., 2018; Li et al., 2018).”
Improving Robustness by Augmenting Training Sentences with Predicate-Argument Structures (2020, methodology): “Strubell et al. (2018) use linguistic information, i.e., dependency parse, part-of-speech tags, and predicates for SRL using a transformer-based encoder (Vaswani et al., 2017).”
SG-Net: Syntax-Guided Machine Reading Comprehension (2019, background): “A related work is from Strubell et al. (2018), which proposed to incorporate syntax with multi-task learning for semantic role labeling.”
Unsupervised Recurrent Neural Network Grammars (2019, background): “Strubell et al. (2018) ﬁnd that supervising intermediate attention layers with syntactic heads improves semantic role labeling, while Shi et al. (2018) observe that for text classiﬁcation, syntactic trees only have marginal impact.”
Jointly Learning to Align and Translate with Transformer Models (2019, methodology): “Augmenting the task objective with linguistic information, such as word alignments, also has had applications beyond MT. Strubell et al. (2018) showed that adding linguistic information from parse trees into one of the attention heads of the transformer model can help in the semantic role labeling.”
Graph-based Knowledge Distillation by Multi-head Attention Network (2019, background): “Because of this useful property, AN has shown meaningful achievements in various signal processing and computer vision ﬁelds [26, 27, 28].”