Cross-Linguistic Syntactic Evaluation of Word Prediction Models

By Aaron Mueller, Garrett Nicolai, Panayiota Petrou-Zeniou, N. Talmina, Tal Linzen. 2020

72 citations recorded by the source corpus.

A range of studies have concluded that neural word prediction models can distinguish grammatical from ungrammatical sentences with high accuracy. However, these studies are based primarily on monolingual evidence from English. To investigate how these models’ ability to learn syntax varies by language, we introduce CLAMS (Cross-Linguistic Assessment of Models on Syntax), a syntactic evaluation suite for monolingual and multilingual models. CLAMS includes subject-verb agreement challenge sets for English, French, German, Hebrew and Russian, generated from grammars we develop. We use CLAMS to evaluate LSTM language models as well as monolingual and multilingual BERT. Across languages, monolingual LSTMs achieved high accuracy on dependencies without attractors, and generally poor accuracy on agreement across object relative clauses. On other constructions, agreement accuracy was generally higher in languages with richer morphology. Multilingual models generally underperformed monolingual models. Multilingual BERT showed high syntactic accuracy on English, but noticeable deficiencies in other languages.

Subfields: Artificial Intelligence, Computational Linguistics & Semantics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yonatan Belinkov
2. Graham Neubig
3. Sebastian Gehrmann
4. Yulia Tsvetkov
5. Antonios Anastasopoulos
6. Yoav Goldberg
7. Tal Linzen
8. Mohit Bansal

In Computational Linguistics & Semantics

1. Yulia Tsvetkov
2. Stuart M. Shieber
3. Graham Neubig
4. Yonatan Belinkov
5. Tal Linzen
6. Sebastian Gehrmann

Citations in the corpus

• Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models (2021, background): “…have studied whether models prefer grammatical completions in subjectverb agreement contexts (Marvin and Linzen, 2018; van Schijndel et al., 2019; Goldberg, 2019; Mueller et al., 2020; Lakretz et al., 2021; Futrell et al., 2019), as well as in filler-gap dependencies (Wilcox et al., 2018, 2019).”
• Evaluating the Morphosyntactic Well-formedness of Generated Texts (2021, background): “Most of these methods, with the exception of Mueller et al. (2020), focus only on English or translation to/from English.”