Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction

By Taeuk Kim, Jihun Choi, Daniel Edmiston, Sang-goo Lee. 2020

93 citations recorded by the source corpus.

With the recent success and popularity of pre-trained language models (LMs) in natural language processing, there has been a rise in efforts to understand their inner workings. In line with such interest, we propose a novel method that assists us in investigating the extent to which pre-trained LMs capture the syntactic notion of constituency. Our method provides an effective way of extracting constituency trees from the pre-trained LMs without training. In addition, we report intriguing findings in the induced trees, including the fact that pre-trained LMs outperform other approaches in correctly demarcating adverb phrases in sentences.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Pre-Trained Models: Past, Present and Future (2021, background): “Kim et al. (2020) propose to leverage the concept of Syntactic Distance to construct the constituency trees of sentences from word representations.”
Pre-trained models for natural language processing: A survey (2020, methodology): “[85] proposed several methods to extract dependency trees and constituency trees from BERT, which proved the BERT’s ability to encode syntax structure.”
Analyzing Individual Neurons in Pre-trained Language Models (2020, unknown): “Previous work on analyzing deep neural networks analyzed how individual layers contribute towards a downstream task (Liu et al., 2019; Kim et al., 2020; Belinkov et al., 2020).”
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning (2021, unknown): “, syntactic [35, 11]) by evaluating the performance of simple classifiers, or probes, on the representations [7, 12, 34, 27, 15].”
Analyzing Redundancy in Pretrained Transformer Models (2020, background): “A number of studies have analyzed representations at layer-level (Conneau et al., 2018; Liu et al., 2019; Tenney et al., 2019; Kim et al., 2020; Belinkov et al., 2020) and at neuron-level (Bau et al., 2019; Dalvi et al., 2019; Suau et al., 2020; Durrani et al., 2020).”