Quantifying the Visual Concreteness of Words and Topics in Multimodal Datasets

By Jack Hessel, David Mimno, Lillian Lee. 2018

44 citations recorded by the source corpus.

Multimodal machine learning algorithms aim to learn visual-textual correspondences. Previous work suggests that concepts with concrete visual manifestations may be easier to learn than concepts with abstract ones. We give an algorithm for automatically computing the visual concreteness of words and topics within multimodal datasets. We apply the approach in four settings, ranging from image captions to images/text scraped from historical books. In addition to enabling explorations of concepts in multimodal datasets, our concreteness scores predict the capacity of machine learning algorithms to learn textual/visual relationships. We find that 1) concrete concepts are indeed easier to learn; 2) the large number of algorithms we consider have similar failure cases; 3) the precise positive relationship between concreteness and performance varies between datasets. We conclude with recommendations for using concreteness scores to facilitate future multimodal research.

Subfields: Computer Vision and Pattern Recognition, Experimental and Cognitive Psychology, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Yejin Choi
2. Jianfeng Gao
3. Rowan Zellers
4. Mohit Bansal
5. Ronan Le Bras
6. Luc Van Gool
7. Dacheng Tao
8. Yonatan Bisk

In Experimental and Cognitive Psychology

1. Yejin Choi
2. Jack Hessel
3. Yonatan Bisk
4. Louis-philippe Morency

In Representation & Self-Supervised Learning

1. Yejin Choi
2. Radu Soricut
3. Jianfeng Gao
4. Zhenhai Zhu
5. Bo Pang
6. Rowan Zellers
7. Jack Hessel
8. Yonatan Bisk

Citations in the corpus

• PIQA: Reasoning about Physical Commonsense in Natural Language (2019, methodology): “Image data also allows for studying the concrete-ness of nouns and provides a natural path forward for further investigation (Hessel, Mimno, and Lee 2018).”
• Beyond Instructional Videos: Probing for More Diverse Visual-Textual Grounding on YouTube (2020, background): “…builds upon prior operational notions of grounding: if an algo-rithm is able to consistently predict specific visual-textual relationships, then that relationship is said to be “grounded” (Lu et al., 2008; Berg et al., 2010; Parikh and Grauman, 2011; Hill and Ko-rhonen, 2014; Hessel et al., 2018).”
• Visually Grounded Neural Syntax Acquisition (2019, unknown): “14 Similarly to VG-NSL, the multimodal concreteness score (Hessel et al., 2018) is also estimated on the MSCOCO training set, using an open-sourced implementation.”
• Separating Skills and Concepts for Novel Visual Question Answering (2021, unknown): “While this can be done with different methods [13, 21, 32], we simply use heuristics.”