Inducing Syntactic Trees from BERT Representations
By Rudolf Rosa, David Marecek. 2019
24 citations recorded by the source corpus.
We use the English model of BERT and explore how a deletion of one word in a sentence changes representations of other words. Our hypothesis is that removing a reducible word (e.g. an adjective) does not affect the representation of other words so much as removing e.g. the main verb, which makes the sentence ungrammatical and of "high surprise" for the language model. We estimate reducibilities of individual words and also of longer continuous phrases (word n-grams), study their syntax-related properties, and then also use them to induce full dependency trees.
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
Citations in the corpus
- Pre-Trained Models: Past, Present and Future (2021, background): “Rosa and Mare ˇ cek (2019) analyze how the deletion of one word in a sentence changes representations of other words to reveal the influence of one word on other words.”
- Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction (2020, background): “…distributions of the pre-trained LMs, while the another group of publications (Mareček & Rosa (2018); Goldberg (2019); Hewitt & Manning (2019); Liu et al. (2019a); Rosa & Mareček (2019), to name a few) delve into the question of the existence of syntactic knowledge in Transformer-based models.”
- Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT (2020, methodology): “Rosa and Mareˇcek (2019) utilize the original MLM objec-tive to estimate each word’s “reducibility” and import simple heuristics into a right-chain baseline to construct dependency trees.”