Grammatical Analysis of Pretrained Sentence Encoders with Acceptability Judgments
By Alex Warstadt, Samuel R. Bowman. 2019
24 citations recorded by the source corpus.
Recent pretrained sentence encoders achieve state of the art results on language understanding tasks, but does this mean they have implicit knowledge of syntactic structures? We introduce a grammatically annotated development set for the Corpus of Linguistic Acceptability (CoLA; Warstadt et al., 2018), which we use to investigate the grammatical knowledge of three pretrained encoders, including the popular OpenAI Transformer (Radford et al., 2018) and BERT (Devlin et al., 2018). We fine-tune these encoders to do acceptability classification over CoLA and compare the models' performance on the annotated analysis set. Some phenomena, e.g. modification by adjuncts, are easy to learn for all models, while others, e.g. long-distance movement, are learned effectively only by models with strong overall performance, and others still, e.g. morphological agreement, are hardly learned by any model.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Samuel R. Bowman
- Mohit Bansal
- Hinrich Schütze
- Yonatan Belinkov
- Percy Liang
- Luke S. Zettlemoyer
- Yejin Choi
- Jianfeng Gao
Citations in the corpus
- Investigating BERT’s Knowledge of Language: Five Analysis Methods with NPIs (2019, methodology): “Warstadt et al. (2018) train models on this task using the CoLA corpus of acceptability judgments as a method for evaluating domain general grammatical knowledge, and Warstadt and Bowman (2019) analyze how these domain general classifiers perform on specific linguistic phenomena.”
- Predicting Inductive Biases of Pre-Trained Models (2021, methodology): “To do this, we fine-tune models for the linguistic acceptability task, a simple sequence classification task as defined in Warstadt & Bowman (2019), longer blocks, using classifiers that are trained on previously transmitted data.”
- Understanding by Understanding Not: Modeling Negation in Language Models (2021, background): “Extensive literature looks at the linguistic knowledge learned by language models (McCoy et al., 2019; Jumelet and Hupkes, 2018; Gulordava et al., 2018; Marvin and Linzen, 2018; Tenney et al., 2019; Warstadt and Bowman, 2019; Talmor et al., 2019).”
- Negated LAMA: Birds cannot fly (2019, background): “There is a wide range of literature analyzing linguistic knowledge stored in pretrained embed-dings (Jumelet and Hupkes, 2018; Gulordava et al., 2018; Giulianelli et al., 2018; McCoy et al., 2019; Dasgupta et al., 2018; Martin and Linzen, 2018; Warstadt and Bowman, 2019; Kann et al., 2019).”