Is Supervised Syntactic Parsing Beneficial for Language Understanding Tasks? An Empirical Investigation

By Goran Glavas, Ivan Vulic. 2020

73 citations recorded by the source corpus.

Traditional NLP has long held (supervised) syntactic parsing necessary for successful higher-level semantic language understanding (LU). The recent advent of end-to-end neural models, self-supervised via language modeling (LM), and their success on a wide range of LU tasks, however, questions this belief. In this work, we empirically investigate the usefulness of supervised parsing for semantic LU in the context of LM-pretrained transformer networks. Relying on the established fine-tuning paradigm, we first couple a pretrained transformer with a biaffine parsing head, aiming to infuse explicit syntactic knowledge from Universal Dependencies treebanks into the transformer. We then fine-tune the model for LU tasks and measure the effect of the intermediate parsing training (IPT) on downstream LU task performance. Results from both monolingual English and zero-shot language transfer experiments (with intermediate target-language parsing) show that explicit formalized syntax, injected into transformers through IPT, has very limited and inconsistent effect on downstream LU performance. Our results, coupled with our analysis of transformers’ representation spaces before and after intermediate parsing, make a significant step towards providing answers to an essential question: how (un)availing is supervised parsing for high-level semantic natural language understanding in the era of large neural models?

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Pre-Trained Models: Past, Present and Future (2021, methodology): “Representation probing have been widely applied to analyze NLP neural models from word embeddings to PTMs (Köhn, 2015; Ettinger et al., 2016; Shi et al., 2016; Adi et al., 2017; Conneau et al., 2018a; Hewitt and Manning, 2019; Glavaš and Vulić, 2021).”
XHate-999: Analyzing and Detecting Abusive Language Across Domains and Languages (2020, unknown): “, 2020) as well as additional target language pretraining (Ponti et al., 2020; Glavaš and Vulić, 2020a), we propose to continue training mBERT and XLM-R via masked language modeling (MLM) (i.”
From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers (2020, background): “(3)We leave an even more general analysis that combines transfer both across tasks (Pruksachatkun et al., 2020; Glavaš and Vulić, 2020) and across languages for future work.”
Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders (2021, unknown): “This finding further confirms a general hypothesis from prior work (Liu et al., 2020; Ben-Zaken et al., 2020; Glavaš and Vulić, 2020) that fine-tuning exposes the wealth of (semantic) knowledge stored in the MLMs for a particular application.”
How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models (2020, methodology): “For UDP, we use a transformer-based variant (Glavaš and Vulić, 2021) of the standard deep biaffine attention dependency parser (Dozat and Manning, 2017).”
MAD-G: Multilingual Adapter Generation for Efficient Cross-Lingual Transfer (2021, background): “variant (Glavaš and Vulić, 2021) of the biaffine dependency parser of Dozat and Manning (2017).”