Parsing as Pretraining

By David Vilares, Michalina Strzyz, Anders Søgaard, Carlos Gómez-Rodríguez. 2020

34 citations recorded by the source corpus.

Recent analyses suggest that encoders pretrained for language modeling capture certain morpho-syntactic structure. However, probing frameworks for word vectors still do not report results on standard setups such as constituent and dependency parsing. This paper addresses this problem and does full parsing (on English) relying only on pretraining architectures – and no decoding. We first cast constituent and dependency parsing as sequence tagging. We then use a single feed-forward layer to directly map word vectors to labels that encode a linearized tree. This is used to: (i) see how far we can reach on syntax modelling with just pretrained encoders, and (ii) shed some light about the syntax-sensitivity of different word vectors (by freezing the weights of the pretraining network during training). For evaluation, we use bracketing F1-score and las, and analyze in-depth differences across representations for span lengths and dependency displacements. The overall results surpass existing sequence tagging parsers on the ptb (93.5%) and end-to-end en-ewt ud (78.8%).

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Xu Han
2. Xiao Liu
3. Ning Ding
4. Zhiyuan Liu
5. Minlie Huang
6. Yuxian Gu
7. Xipeng Qiu
8. Zhengyan Zhang

Citations in the corpus

• Pre-Trained Models: Past, Present and Future (2021, background): “Similarly, several works also reveal the strong syntax encoding of PTMs (Vilares et al., 2020; Warstadt and Bowman, 2020; Hewitt and Manning, 2019).”
• A Primer in BERTology: What We Know About How BERT Works (2020, background): “Enough syntactic information seems to be captured in the token embeddings themselves to recover syntactic trees (Vilares et al., 2020; Kim et al., 2020; Rosa and Mareček, 2019), although probing classifiers could not recover the labels of distant parent nodes in the syntactic tree (Liu et al.”
• Parallel Trees: a novel resource with aligned dependency and constituency syntactic representations (2025, unknown): “BERT and ELMO were also used by Vilares et al. (2020) to compare the effectiveness of precomputed embeddings for dependency and constituency parsing.”