Multilingual Constituency Parsing with Self-Attention and Pre-Training

By Nikita Kitaev, Steven Cao, Daniel Klein. 2018

287 citations recorded by the source corpus.

We show that constituency parsing benefits from unsupervised pre-training across a variety of languages and a range of pre-training conditions. We first compare the benefits of no pre-training, fastText, ELMo, and BERT for English and find that BERT outperforms ELMo, in large part due to increased model capacity, whereas ELMo in turn outperforms the non-contextual fastText embeddings. We also find that pre-training is beneficial across all 11 languages tested; however, large model sizes (more than 100 million parameters) make it computationally expensive to train separate models for each language. To address this shortcoming, we show that joint multilingual pre-training and fine-tuning allows sharing all but a small number of parameters between ten languages in the final model. The 10x reduction in model size compared to fine-tuning one model per language causes only a 3.2% relative error increase in aggregate. We further explore the idea of joint fine-tuning and show that it gives low-resource languages a way to benefit from the larger datasets of other languages. Finally, we demonstrate new state-of-the-art results for 11 languages, including English (95.8 F1) and Chinese (91.8 F1).

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (2019, background): “The tasks that have proven amenable to this general approach include question answering, sentiment analysis, textual entailment, and parsing, among many others (Devlin et al., 2019; Kitaev and Klein, 2018).”
Self-Instruct: Aligning Language Models with Self-Generated Instructions (2022, methodology): “We use the Berkeley Neural Parser7 (Kitaev and Klein, 2018; Kitaev et al., 2019) to parse the instructions and then extract the verb that is closest to the root as well as its first direct noun object.”
Pre-Trained Language Models for Interactive Decision-Making (2022, methodology): “Successful uses of representations from pretrained models include syntactic parsing (Kitaev et al., 2018) and language-to-code translation (Wang et al.”
FlauBERT: Unsupervised Language Model Pre-training for French (2019, methodology): “Model description We use the parser described by Kitaev and Klein (2018) and Kitaev et al. (2019).”
Head-Driven Phrase Structure Grammar Parsing on Penn Treebank (2019, methodology): “We use the same training settings as (Kitaev and Klein, 2018a) and (Kitaev and Klein, 2018b).”
Harnessing the Power of David against Goliath: Exploring Instruction Data Generation without Using Closed-Source Models (2023, methodology): “(2022a), we use the Berkeley Neural Parser (Kitaev and Klein, 2018; Kitaev et al., 2019) to extract the top 20 most common root verbs and their top 4 direct noun objects of our generated instruction data, which are shown in Figure 5.”
Learning Syntax from Naturally-Occurring Bracketings (2021, methodology): “(4)For “ground-truth” structures in the Wikipedia data, we apply a state-of-the-art PTB-trained constituency parser (Kitaev et al., 2019).”
IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP (2020, background): “…have quickly become ubiquitous in NLP and driven substantial empirical gains across tasks including NER (Devlin et al., 2019), POS tagging (Devlin et al., 2019), single document summarization (Liu and Lapata, 2019), syntactic parsing (Kitaev et al., 2019), and discourse analysis (Nie et al., 2019).”
Cross-Domain Generalization of Neural Constituency Parsers (2019, methodology): “2 and 3) to a model that uses a pre-2 trained BERT encoder (Kitaev et al., 2019), using the publicly-released code 3 to train and evaluate both models.”
Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little (2021, methodology): “, 2020), but did not observe any difference in UAS with different pre-trained models (see Appendix G) (11)PTB data (Kitaev et al., 2019) is used from github.”