Constituency Parsing with a Self-Attentive Encoder

By Nikita Kitaev, Daniel Klein. 2018

596 citations recorded by the source corpus.

We demonstrate that replacing an LSTM encoder with a self-attentive architecture can lead to improvements to a state-of-the-art discriminative constituency parser. The use of attention makes explicit the manner in which information is propagated between different locations in the sentence, which we use to both analyze our model and propose potential improvements. For example, we find that separating positional and content information in the encoder can lead to improved parsing accuracy. Additionally, we evaluate different approaches for lexical representation. Our parser achieves new state-of-the-art results for single models trained on the Penn Treebank: 93.55 F1 without the use of any external data, and 95.13 F1 when using pre-trained word representations. Our parser also outperforms the previous best-published accuracy figures on 8 of the 9 languages in the SPMRL dataset.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

Dissecting Contextual Word Embeddings: Architecture and Representation (2018, background): “In addition to machine translation, it has also provided strong results for Penn Treebank constituency parsing (Kitaev and Klein, 2018) and semantic role labeling (Tan et al.”
Universal Transformers (2018, background): “Self-attentive feed-forward sequence models have been shown to achieve impressive results on sequence modeling tasks including machine translation [31], image generation [30] and constituency parsing [18], thereby presenting a compelling alternative to recurrent neural networks (RNNs) which has rema”
Cloze-driven Pretraining of Self-attention Networks (2019, positive): “We adopted the current state-of-the-art architecture (Kitaev and Klein, 2018).”
Self-Instruct: Aligning Language Models with Self-Generated Instructions (2022, methodology): “We use the Berkeley Neural Parser7 (Kitaev and Klein, 2018; Kitaev et al., 2019) to parse the instructions and then extract the verb that is closest to the root as well as its first direct noun object.”
The Effect of Natural Distribution Shift on Question Answering Models (2020, methodology): “Following Rajpurkar et al. (2016), we automatically categorize each answer into numerical and non-numerical answers, named entities, and constituents us-ing spaCy (Honnibal & Montani, 2017) and the constituency parser from Kitaev & Klein (2018).”
SELFEXPLAIN: A Self-Explaining Architecture for Neural Text Classifiers (2021, methodology): “We generate parse trees (Kitaev and Klein, 2018) to extract target concepts for the input and follow same pre-processing steps as the original encoder conﬁgurations for the rest.”
LongForm: Effective Instruction Tuning with Reverse Instructions (2023, positive): “Diversity: Following Wang et al. (2023b), we analyze the diversity of the generated instructions by parsing each instruction in the corpus subset with Berkeley Neural Parser (Kitaev and Klein, 2018).”
What do you learn from context? Probing for sentence structure in contextualized word representations (2019, methodology): “Recent work has consistently demonstrated the strong empirical performance of contextualized word representations, including CoVe (McCann et al., 2017), ULMFit (Howard & Ruder, 2018), ELMo (Peters et al., 2018a; Lee et al., 2018; Strubell et al., 2018; Kitaev & Klein, 2018).”
Direct Output Connection for a High-Rank Language Model (2018, neutral): “We expect that we can achieve even better score by replacing the base parser with the current state-of-the-art one (Kitaev and Klein, 2018).”
Episodic Memory in Lifelong Language Learning (2019, background): “In language learning, progress in unsupervised pretraining (Peters et al., 2018; Howard & Ruder, 2018; Devlin et al., 2018) has driven advances in many language understanding tasks (Kitaev & Klein, 2018; Lee et al., 2018).”
Do Attention Heads in BERT Track Syntactic Dependencies? (2019, background): “BERT and BERTbased models significantly improve the state-ofthe-art on many tasks such as constituency parsing (Kitaev and Klein, 2018), question answering (Rajpurkar et al.”
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization (2020, methodology): “Speciﬁcally, we mask important text spans in a sentence, including noun phrases extracted by a constituency parser (Kitaev and Klein, 2018) and named entities extracted by the Stanford CoreNLP NER model (Finkel et al., 2005; Manning et al., 2014).”
Analyzing Dynamic Adversarial Training Data in the Limit (2021, methodology): “6 To compute Yngve scores, we parse sentences using the Benepar parser (Kitaev and Klein, 2018) based on T5 small (Raffel et al.”
Multilingual Constituency Parsing with Self-Attention and Pre-Training (2018, methodology): “Our joint multilingual model outperforms treebankonly models (Kitaev and Klein, 2018) for each of the languages (88.”
Constrained Decoding for Neural NLG from Compositional Representations in Task-Oriented Dialogue (2019, methodology): “Finally, we used the Berkeley neural parser (Kitaev and Klein, 2018) to identify spans coordinated by but, and added CONTRAST relations as parents of the coordinated arguments.”
FlauBERT: Unsupervised Language Model Pre-training for French (2019, methodology): “Without pre-training, we replicate the result from Kitaev and Klein (2018).”
Parsing as Pretraining (2020, methodology): “For constituents, BERT obtains a 93.5% F1-score vs the 95.8% reported by Kitaev and Klein (2018b) and Zhou and Zhao (2019), which are the current best performing models and use BERT representations as input (together with PoS tag embedddings, and in the case of Zhou and Zhao also char embeddings).”
Multi-Step Inference for Reasoning over Paragraphs (2020, methodology): “…et al., 2019; Yang et al., 2019), most of NLP benchmarks got new state-of-the-art results by the models built beyond the pre-trained transformer on specific tasks (e.g. syntactic parsing, semantic parsing and GLUE) (Wang et al., 2018; Kitaev and Klein, 2018; Zhang et al., 2019; Tsai et al., 2019).”
Distribution Augmentation for Low-Resource Expressive Text-To-Speech (2022, unknown): “For constituency parsing we use Berkeley Neural Parser [26].”