COGS: A Compositional Generalization Challenge Based on Semantic Interpretation

By Najoung Kim, Tal Linzen. 2020

346 citations recorded by the source corpus.

Natural language is characterized by compositionality: the meaning of a complex expression is constructed from the meanings of its constituent parts. To facilitate the evaluation of the compositional abilities of language processing architectures, we introduce COGS, a semantic parsing dataset based on a fragment of English. The evaluation portion of COGS contains multiple systematic gaps that can only be addressed by compositional generalization; these include new combinations of familiar syntactic structures, or new combinations of familiar words and familiar structures. In experiments with Transformers and LSTMs, we found that in-distribution accuracy on the COGS test set was near-perfect (96--99%), but generalization accuracy was substantially lower (16--35%) and showed high sensitivity to random seed ($\pm$6--8%). These findings indicate that contemporary standard NLP models are limited in their compositional generalization capacity, and position COGS as a good way to measure progress.

Subfields: Artificial Intelligence, Language and Linguistics, Evaluation & Benchmarks, Computational Linguistics & Semantics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Luke S. Zettlemoyer
  3. Yejin Choi
  4. Percy Liang
  5. Jianfeng Gao
  6. Douwe Kiela
  7. Noah A. Smith
  8. Graham Neubig

In Language and Linguistics

  1. Ellie Pavlick
  2. Ming-Wei Chang
  3. Panupong Pasupat
  4. Mohit Bansal
  5. Benjamin Van Durme
  6. Christopher Potts
  7. Robin Jia
  8. Douwe Kiela

In Evaluation & Benchmarks

  1. Mohit Bansal
  2. Douwe Kiela
  3. Robin Jia
  4. Luke S. Zettlemoyer
  5. Sebastian Riedel
  6. Samuel R. Bowman
  7. Matt Gardner
  8. Ellie Pavlick

In Computational Linguistics & Semantics

  1. Panupong Pasupat
  2. Peter Shaw
  3. Kristina Toutanova
  4. Ming-Wei Chang
  5. Xinyun Chen
  6. Mohit Bansal
  7. Christopher Potts
  8. Olivier Bousquet

Citations in the corpus