Measuring Compositional Generalization: A Comprehensive Method on Realistic Data

By Daniel Keysers, Nathanael Scharli, Nathan Scales, Hylke Buisman, Daniel Furrer, S. Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon and 4 more. 2019

421 citations recorded by the source corpus.

State-of-the-art machine learning methods exhibit limited compositional generalization. At the same time, there is a lack of realistic benchmarks that comprehensively measure this ability, which makes it challenging to find and evaluate improvements. We introduce a novel method to systematically construct such benchmarks by maximizing compound divergence while guaranteeing a small atom divergence between train and test sets, and we quantitatively compare this method to other approaches for creating compositional generalization benchmarks. We present a large and realistic natural language question answering dataset that is constructed according to this method, and we use it to analyze the compositional generalization ability of three machine learning architectures. We find that they fail to generalize compositionally and that there is a surprisingly strong negative correlation between compound divergence and accuracy. We also demonstrate how our method can be used to create new compositionality benchmarks on top of the existing SCAN dataset, which confirms these findings.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Augmented Language Models: a Survey (2023, background): “This recursive approach slightly differs from CoT since the latter does not explicitly formulate sub-problems, and can be especially useful for complex tasks, given that compositional generalization can be challenging for LMs (Lake and Baroni, 2018; Keysers et al., 2019; Li et al., 2022a).”
Least-to-Most Prompting Enables Complex Reasoning in Large Language Models (2022, background): “For instance, there might only be examples of reversing short arrays in training while the model is asked to reverse long arrays in testing; (2) Compositional generalization (Lake & Baroni, 2018; Keysers et al., 2020), where a model is exposed to a set of components during training and needs to gene”
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, background): “This is driven by the compositional generality of language (Hupkes et al., 2020; Keysers et al., 2019), and can be applied to domains in which data is scarce, models are only available through APIs without source-code access, or it may be prohibitively expensive to train a new large multimodal model”
CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning (2020, background): “This encourages models to reason about a potentially inﬁnite number of novel combinations of familiar concepts – an ability believed to be a limitation of current AI systems (Lake and Baroni, 2017; Keysers et al., 2020).”
On the Binding Problem in Artificial Neural Networks (2020, background): “For example, it is difficult for neural language models to generalize syntactic rules such as verb tenses or embedded clauses in a systematic manner (Keysers et al., 2020; Lake and Baroni, 2018; Loula et al., 2018; Hupkes et al., 2020).”
Measuring and Narrowing the Compositionality Gap in Language Models (2022, background): “Keysers et al. (2020) generate compositional questions to test small encoder-decoder models, but they do not test pretrained LMs.”
Neurosymbolic Programming (2021, background): “SQL for a relational database or SPARQL for a knowledge graph) are effective, they often fail to inductively generalize to different parameterizations of the same question or exhibit other compositional generalization (Keysers et al., 2020).”
A Benchmark for Systematic Generalization in Grounded Language Understanding (2020, background): “Lake and Baroni [25] proposed the SCAN benchmark for evaluating systematic generalization, which was distinguished by testing for linguistic generalization and learning abstract compositional rules (see also [7, 3, 22, 19]).”
Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing (2022, methodology): “CFQ The CFQ dataset (Keysers et al., 2020) contains questions paired with SPARQL queries.”
Learning to Recombine and Resample Data for Compositional Generalization (2020, background): “For example, learners should generalize from actions seen in isolation to more complex commands involving those actions (Lake et al., 2019), and from relations of the form r(a,b) to r(b,a) (Keysers et al., 2020; Bahdanau et al., 2019b).”
Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both? (2020, background): “Other synthetic datasets have been developed to evaluate aspects of compositional generalization beyond SCAN, including NACS (Bast-ings et al., 2018), CFQ (Keysers et al., 2020), and COGS (Kim and Linzen, 2020).”
Visually Grounded Continual Learning of Compositional Semantics (2020, background): “It implies learning the compositionally generalize, i.e. , learning to identify atomic concepts in an example and combine them (Keysers et al., 2020) is crucial for performance.”
Compositional Generalization via Neural-Symbolic Stack Machines (2020, background): “In contrast, deep neural networks, including the state-of-the-art models for natural language understanding, typically lack such generalization abilities [24, 22, 30], although they have demonstrated impressive performance on various applications.”
Towards Ecologically Valid Research on Language User Interfaces (2020, neutral): “For example, some questions in the Compositional Freebase Questions (CFQ) dataset (Keysers et al., 2019) are hard to understand because of the conjunction of many noun or verb phrases: Did Patrick Scully’s sibling marry Carolyn Zeifman, inﬂuence Tetsuo II: Body Hammer’s art director, director, and…”
Case-based Reasoning for Natural Language Queries over Knowledge Bases (2021, methodology): “The revise step aligns any such missing edges (relations) with existing semantically-similar edges in the KB. tive for questions that need novel combination of KB relations, achieving competitive results on multiple KBQA benchmarks such as WebQuestionsSP (Yih et al., 2016), ComplexWebQuestions (CWQ)”
Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases (2020, background): “Another line of work has been focusing on compositional generalization on a number of specifically-synthesized datasets [21, 23].”
Quantifying Adaptability in Pre-trained Language Models with 500 Tasks (2021, methodology): “Many prior studies of compositionality focus on instance-level compositionality (Lake and Ba-roni, 2018; Keysers et al., 2020): they test whether models can generalize to new instances by combining information from previously-seen instances within the same task .”
LEGO: Latent Execution-Guided Reasoning for Multi-Hop Question Answering on Knowledge Graphs (2021, background): “…parsing requires the ground truth programs for training (Zelle & Mooney, 1996; Zettle-moyer & Collins, 2012; Jia & Liang, 2016; Yu et al., 2018b;a; Keysers et al., 2020), some recent approaches have been proposed to improve the model performance with weak supervision, i.e., the training…”