A Benchmark for Systematic Generalization in Grounded Language Understanding

By Laura Ruis, Jacob Andreas, Marco Baroni, Diane Bouchacourt, Brenden M. Lake. 2020

166 citations recorded by the source corpus.

Human language users easily interpret expressions that describe unfamiliar situations composed from familiar parts ("greet the pink brontosaurus by the ferris wheel"). Modern neural networks, by contrast, struggle to interpret compositions unseen in training. In this paper, we introduce a new benchmark, gSCAN, for evaluating compositional generalization in models of situated language understanding. We take inspiration from standard models of meaning composition in formal linguistics. Going beyond an earlier related benchmark that focused on syntactic aspects of generalization, gSCAN defines a language grounded in the states of a grid world. This allows us to build novel generalization tasks that probe the acquisition of linguistically motivated rules. For example, agents must understand how adjectives such as 'small' are interpreted relative to the current world state or how adverbs such as 'cautiously' combine with new verbs. We test a strong multi-modal baseline model and a state-of-the-art compositional method finding that, in most cases, they fail dramatically when generalization requires systematic compositional rules.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

The MAGICAL Benchmark for Robust Imitation (2020, background): “There are also several instruction-following benchmarks for evaluating generalisation in natural language understanding [29, 34].”
Inductive biases for deep learning of higher-level cognition (2020, background): “However, a form of compositionality that we propose here and should be better incorporated in deep learning is the form called systematicity (Lake and Baroni, 2018) defined by linguists, and more recently systematic generalization in machine learning papers (Bahdanau et al., 2018; Ruis et al., 2020;”
Word meaning in minds and machines (2020, methodology): “After learning that a certain class of heavy objects require more actions to successfully “ push ” them, current models generalize that these same objects require more actions when “ pulling, ” given experience pulling other classes of heavy objects (Ruis et al., 2020).”
A large-scale benchmark for few-shot program induction and synthesis (2021, methodology): “To improve this, more compositional methods to meta-learning are increasingly being proposed (Alet et al., 2018; Bengio et al., 2019; Ke et al., 2019; Mendez & Eaton, 2020; Ruis et al., 2020).”
One-Shot Learning from a Demonstration with Hierarchical Latent Language (2022, unknown): “Length Generalization Neural sequence models show to fail to generalize to task lengths longer than those seen in training (Ruis et al., 2020).”
Learning to Solve Complex Tasks by Talking to Agents (2021, background): “…as discussed above, there has been renewed interest in developing hard synthetic reasoning challenges (Lake and Baroni, 2018; Sinha et al., 2019; Ruis et al., 2020; Clark et al., 2020; Betz and Richard-son, 2021) that help to more systematically identify model strengths and weaknesses and…”
Multimodal Graph Networks for Compositional Generalization in Visual Question Answering (2020, unknown): “Concurrent to this work, Gao et al. [15] also proposed a fused multimodal representation, using iterative message passing between text and visual embeddings for the grounded SCAN task [41].”
Intra-agent speech permits zero-shot task acquisition (2022, unknown): “, “compositional” generalization or systematicity [42]).”
Modularity Improves Out-of-Domain Instruction Following (2020, background): “Second, monolithic models often struggle in compositional generalization : composing atomic parts, such as actions or goals, where the parts are seen in training but their compositions are not (Lake and Baroni, 2018; Ruis et al., 2020; Hill et al., 2020).”
Compositional Generalization via Neural-Symbolic Stack Machines (2020, background): “There has been an increasing interest in studying the compositional generalization of deep neural networks for natural language understanding [24, 22, 4, 39].”
PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World (2021, background): “Past work evaluating these types of zero-shot generalization have also looked into how well models can compose concepts in language together (Lake and Baroni, 2018; Ruis et al., 2020).”
Quantifying Adaptability in Pre-trained Language Models with 500 Tasks (2021, methodology): “These datasets have been used to study long-range agreement (Marvin and Linzen, 2018), compositional generalization (Lake and Ba-roni, 2018; Ruis et al., 2020; Keysers et al., 2020), and mathematical reasoning (Saxton et al., 2019).”
Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoning (2021, methodology): “We use the gSCAN benchmark (Ruis et al., 2020), a recently proposed grounded instruction following dataset designed to measure compositional generalization in neural systems.”
Can Small and Synthetic Benchmarks Drive Modeling Innovation? A Retrospective Study of Question Answering Modeling Approaches (2021, unknown): “…benchmarks only use synthetic data during training and testing for the purpose of comparing modeling approaches (Weston et al., 2016; Lake and Baroni, 2018; Chevalier-Boisvert et al., 2018; Saxton et al., 2019; Kim and Linzen, 2020; Ruis et al., 2020; Hui et al., 2020; Keysers et al., 2020).”
The EOS Decision and Length Extrapolation (2020, background): “Standard evaluations like exact match or BLEU (Papineni et al., 2002) on length-generalization datasets like SCAN and gSCAN (Lake and Baroni, 2018; Ruis et al., 2020) evaluate the ability to perform both components, but models are known to fail because of (ii) alone, so these metrics don’t test (i)…”
Compositional Semantic Parsing with Large Language Models (2022, background): “…learning has been a challenging problem that attracts attention across fields, including vision (Johnson et al., 2017; Bahdanau et al., 2019; Ruis et al., 2020; Nikolaus et al., 2019) and language domains (Lake & Baroni, 2018; Keysers et al., 2020; Kim & Linzen, 2020; Shaw et al., 2021; Yin…”