What do RNN Language Models Learn about Filler–Gap Dependencies?

By Ethan Gotlieb Wilcox, Roger Levy, Takashi Morita, Richard Futrell. 2018

206 citations recorded by the source corpus.

RNN language models have achieved state-of-the-art perplexity results and have proven useful in a suite of NLP tasks, but it is as yet unclear what syntactic generalizations they learn. Here we investigate whether state-of-the-art RNN language models represent long-distance filler–gap dependencies and constraints on them. Examining RNN behavior on experimentally controlled sentences designed to expose filler–gap dependencies, we show that RNNs can represent the relationship in multiple syntactic positions and over large spans of text. Furthermore, we show that RNNs learn a subset of the known restrictions on filler–gap dependencies, known as island constraints: RNNs show evidence for wh-islands, adjunct islands, and complex NP islands. These studies demonstrates that state-of-the-art RNN models are able to learn and generalize about empty syntactic positions.

Subfields: Artificial Intelligence, Natural Language Processing, Computational Linguistics & Semantics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Yonatan Belinkov
  2. Samuel R. Bowman
  3. Mohit Bansal
  4. Yoav Goldberg
  5. Tal Linzen
  6. Percy Liang
  7. Noah A. Smith
  8. Luke S. Zettlemoyer

In Natural Language Processing

  1. Yonatan Belinkov
  2. Samuel R. Bowman
  3. Tal Linzen
  4. Yoav Goldberg
  5. Mohit Bansal
  6. Noah A. Smith
  7. Douwe Kiela
  8. Percy Liang

In Computational Linguistics & Semantics

  1. Yonatan Belinkov
  2. Tal Linzen
  3. Yoav Goldberg
  4. Dieuwke Hupkes
  5. Roger Levy
  6. Peng Qian
  7. Ethan Gotlieb Wilcox
  8. Willem H. Zuidema

Citations in the corpus