COGS: A Compositional Generalization Challenge Based on Semantic Interpretation
By Najoung Kim, Tal Linzen. 2020
346 citations recorded by the source corpus.
Natural language is characterized by compositionality: the meaning of a complex expression is constructed from the meanings of its constituent parts. To facilitate the evaluation of the compositional abilities of language processing architectures, we introduce COGS, a semantic parsing dataset based on a fragment of English. The evaluation portion of COGS contains multiple systematic gaps that can only be addressed by compositional generalization; these include new combinations of familiar syntactic structures, or new combinations of familiar words and familiar structures. In experiments with Transformers and LSTMs, we found that in-distribution accuracy on the COGS test set was near-perfect (96--99%), but generalization accuracy was substantially lower (16--35%) and showed high sensitivity to random seed ($\pm$6--8%). These findings indicate that contemporary standard NLP models are limited in their compositional generalization capacity, and position COGS as a good way to measure progress.
Subfields: Artificial Intelligence, Language and Linguistics, Evaluation & Benchmarks, Computational Linguistics & Semantics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Yejin Choi
- Percy Liang
- Jianfeng Gao
- Douwe Kiela
- Noah A. Smith
- Graham Neubig
In Language and Linguistics
- Ellie Pavlick
- Ming-Wei Chang
- Panupong Pasupat
- Mohit Bansal
- Benjamin Van Durme
- Christopher Potts
- Robin Jia
- Douwe Kiela
In Evaluation & Benchmarks
- Mohit Bansal
- Douwe Kiela
- Robin Jia
- Luke S. Zettlemoyer
- Sebastian Riedel
- Samuel R. Bowman
- Matt Gardner
- Ellie Pavlick
In Computational Linguistics & Semantics
- Panupong Pasupat
- Peter Shaw
- Kristina Toutanova
- Ming-Wei Chang
- Xinyun Chen
- Mohit Bansal
- Christopher Potts
- Olivier Bousquet
Citations in the corpus
- Dynabench: Rethinking Benchmarking in NLP (2021, background): “Challenging test sets have shown that many state-of-the-art NLP models struggle with compositionality (Nie et al., 2019; Kim and Linzen, 2020; Yu and Ettinger, 2020; White et al., 2020), and find it difficult to pass the myriad stress tests for social (Rudinger et al., 2018; May et al., 2019; Nangia…”
- WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, neutral): “This instability in OOD performance has been reported in natural language processing settings (McCoy et al., 2019; Kim and Linzen, 2020) and also in vision and healthcare applications (D’Amour et al., 2020).”
- Can Small and Synthetic Benchmarks Drive Modeling Innovation? A Retrospective Study of Question Answering Modeling Approaches (2021, neutral): “Synthetic benchmarks only use synthetic data during training and testing for the purpose of comparing modeling approaches (Weston et al., 2016; Lake and Baroni, 2018; Chevalier-Boisvert et al., 2018; Saxton et al., 2019; Kim and Linzen, 2020; Ruis et al., 2020; Hui et al., 2020; Keysers et al., 2020”
- Documenting the English Colossal Clean Crawled Corpus (2021, background): “For example, it can hinder testing how well pretrained models perform on new, previously unseen inputs such as new combinations of familiar syntactic structures (Kim and Linzen, 2020).”
- One-Shot Learning from a Demonstration with Hierarchical Latent Language (2022, background): “This builds upon existing work that tests for compositional generalization in language processing models [4, 5, 7], but alters the input modality such that agents must respond to nonverbal demonstrations of novel tasks.”
- Sequence-to-Sequence Learning with Latent Neural Grammars (2021, neutral): “1 However, despite their excellent performance across key benchmarks these models are often sample inefficient and can moreover fail spectacularly on diagnostic tasks designed to test for compositional generalization [68, 63].”
- Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both? (2020, methodology): “Other synthetic datasets have been developed to evaluate aspects of compositional generalization beyond SCAN, including NACS (Bast-ings et al., 2018), CFQ (Keysers et al., 2020), and COGS (Kim and Linzen, 2020).”
- Are Pretrained Convolutions Better than Pretrained Transformers? (2021, background): “…another model architecture variation by studying the power of convolutional neural network as the backbone of pre-trained models for NLP. Convolutions have always been an interesting choice for sequence modeling and NLP applications (Kim, 2014; Bai et al., 2018; Kalchbrenner et al., 2016).”
- Prompting GPT-3 To Be Reliable (2022, neutral): “…in the training data which fail on counter test examples (McCoy et al., 2019; Zhang et al., 2019; Poliak et al., 2018; Gururangan et al., 2018); LLMs also fail on new compositional structures that are not observed during traing (Kim & Linzen, 2020; Keysers et al., 2020; Bogin et al., 2022).”
- Quantifying Adaptability in Pre-trained Language Models with 500 Tasks (2021, background): “…to past work that uses synthetic data to characterize LM performance on single examples (Weston et al., 2016; Lake and Baroni, 2018; Saxton et al., 2019; Kim and Linzen, 2020; Keysers et al., 2020; Liu et al., 2021a), T ASK B ENCH 500 enables systematic study of LM adaptability at the task level.”
- How Much Do Language Models Copy From Their Training Data? Evaluating Linguistic Novelty in Text Generation Using RAVEN (2021, neutral): “It is often discussed in the context of out-of-distribution generalization (Hadley, 1994; Hupkes et al., 2020; Keysers et al., 2020; Li et al., 2021), typically relying on synthetic datasets to test models’ compositional abilities (Lake and Baroni, 2018; Kim and Linzen, 2020; McCoy et al., 2020).”
- Compositional Semantic Parsing with Large Language Models (2022, methodology): “Similarly to symbolic grammar learning techniques on SCAN, these approaches achieve impressive performance on several benchmarks, and represent the previous state of the art on CFQ (Keysers et al., 2020) and COGS (Kim & Linzen, 2020).”
- Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing (2022, neutral): “More details on all datasets and splits are in Appendix A. COGS The COGS dataset (Kim and Linzen, 2020) contains sentences paired with logical forms.”
- Frequency Effects on Syntactic Rule Learning in Transformers (2021, background): “Our formulation of the SVA task also relates to work which investigates neural networks’ abilities to learn lexical abstractions (Chronis and Erk, 2020; Kim and Smolensky, 2021) and to reason systematically (Lake and Baroni, 2018; Yanaka et al., 2019; Kim and Linzen, 2020; Goodwin et al., 2020).”