ShapeWorld - A new test methodology for multimodal language understanding

By Alexander Kuhnle, Ann A. Copestake. 2017

76 citations recorded by the source corpus.

We introduce a novel framework for evaluating multimodal deep learning models with respect to their language understanding and generalization abilities. In this approach, artificial data is automatically generated according to the experimenter's specifications. The content of the data, both during training and evaluation, can be controlled in detail, which enables tasks to be created that require true generalization abilities, in particular the combination of previously introduced concepts in novel ways. We demonstrate the potential of our methodology by evaluating various visual question answering models on four different tasks, and show how our framework gives us detailed insights into their capabilities and limitations. By open-sourcing our framework, we hope to stimulate progress in the field of multimodal language understanding.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Evaluation & Benchmarks

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “Others have investigated how grounded language can be used to improve visual representations and classifiers on the ShapeWorld dataset (Kuhnle & Copestake, 2017; Andreas et al., 2017; Mu et al., 2019).”
Shaping Visual Representations with Language for Few-Shot Classification (2019, positive): “First, we use the ShapeWorld [20] dataset devised by [1], which consists of 9000 training, 1000 validation, and 4000 test tasks (Figure 2).”
A Corpus for Reasoning about Natural Language Grounded in Photographs (2018, background): “Recently, the relatively simple language and reasoning in existing resources motivated datasets that focus on compositional language, mostly using synthetic data for language and vision (Andreas et al., 2016; Johnson et al., 2017a; Kuhnle and Copestake, 2017; Kahou et al., 2018; Yang et al., 2018).”
A Benchmark for Systematic Generalization in Grounded Language Understanding (2020, background): “This result conﬁrms results in related work that vanilla neural networks fall short in recombining familiar concepts in novel ways (Kuhnle & Copestake, 2017; Loula et al., 2018).”
A dataset and architecture for visual reasoning with a working memory (2018, background): “attribute identiication, counting, comparison, multiple attention, and logical operations) [10, 42, 43, 13, 44].”
MMBench: Is Your Multi-modal Model an All-around Player? (2023, background): “…19 0.6% PISC [29] customize & selection 15 0.5% Places [54] customize & selection 59 1.8% ScienceQA [34] customize & selection 156 4.8% ShapeWorld [25] customize & selection 20 0.6% TextVQA [42] customize & selection 18 0.6% VSR [31] customize & selection 19 0.6% W3C School [1] customize 20 0.6%…”
Emergent Communication of Generalizations (2021, methodology): “We use the ShapeWorld visual reasoning dataset [18] (Figure 2, left).”
What's "up" with vision-language models? Investigating their struggle with spatial reasoning (2023, background): “Spatial reasoning has long been evaluated by vision-language benchmarks: VQAv2 (Goyal et al., 2016), GQA (Hudson and Manning, 2019), NLVR2 (Suhr et al., 2018), CLEVR (Johnson et al., 2017) and ShapeWorld (Kuhnle and Copestake, 2017) all contain questions requiring spatial reasoning.”