Refining Targeted Syntactic Evaluation of Language Models

By Benjamin Newman, Kai-Siang Ang, Julia Gong, John Hewitt. 2021

52 citations recorded by the source corpus.

Targeted syntactic evaluation of subject-verb number agreement in English (TSE) evaluates language models’ syntactic knowledge using hand-crafted minimal pairs of sentences that differ only in the main verb’s conjugation. The method evaluates whether language models rate each grammatical sentence as more likely than its ungrammatical counterpart. We identify two distinct goals for TSE. First, evaluating the systematicity of a language model’s syntactic knowledge: given a sentence, can it conjugate arbitrary verbs correctly? Second, evaluating a model’s likely behavior: given a sentence, does the model concentrate its probability mass on correctly conjugated verbs, even if only on a subset of the possible verbs? We argue that current implementations of TSE do not directly capture either of these goals, and propose new metrics to capture each goal separately. Under our metrics, we find that TSE overestimates systematicity of language models, but that models score up to 40% better on verbs that they predict are likely in context.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models (2021, background): “Further work should also explore the impact of specific verbs on syntactic agreement mechanisms (Newman et al., 2021).”
When a sentence does not introduce a discourse entity, Transformer-based models still sometimes refer to it (2022, methodology): “In that case, the evaluations in Experiment 2 may underestimate the models’ abilities (Newman et al., 2021) and the results may not be very relevant for practical purposes for which one uses an LM to generate texts.”
What Can a Generative Language Model Answer About a Passage? (2021, background): “In recent LM research that probes the syntactic knowledge in these models, the structured datasets of minimal grammatical/ungrammatical sentence pairs test when LMs assign a higher probability to the grammatical sentence than the ungrammatical one (Marvin and Linzen, 2018; Newman et al., 2021).”
Frequency Effects on Syntactic Rule Learning in Transformers (2021, methodology): “This result concurs with Newman et al. (2021)’s finding that model performance drops when testing verbs that the model finds unlikely.”