Integrating Text and Image: Determining Multimodal Document Intent in Instagram Posts

By Julia Kruk, Jonah Lubin, Karan Sikka, Xiaoyu Lin, Dan Jurafsky, Ajay Divakaran. 2019

116 citations recorded by the source corpus.

Computing author intent from multimodal data like Instagram posts requires modeling a complex relationship between text and image. For example, a caption might evoke an ironic contrast with the image, so neither caption nor image is a mere transcript of the other. Instead they combine—via what has been called meaning multiplication (Bateman et al.)- to create a new meaning that has a more complex relation to the literal meanings of text and image. Here we introduce a multimodal dataset of 1299 Instagram posts labeled for three orthogonal taxonomies: the authorial intent behind the image-caption pair, the contextual relationship between the literal meanings of the image and caption, and the semiotic relationship between the signified meanings of the image and caption. We build a baseline deep multimodal classifier to validate the taxonomy, showing that employing both text and image improves intent detection by 9.6 compared to using only the image modality, demonstrating the commonality of non-intersective meaning multiplication. The gain with multimodality is greatest when the image and caption diverge semiotically. Our dataset offers a new resource for the study of the rich meanings that result from pairing text and image.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Experimental and Cognitive Psychology, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Experimental and Cognitive Psychology

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, methodology): “Other datasets using internet data include Food101 [80], where the goal is to predict the dish of recipes and images; various versions of Yelp reviews [52]; Walmart and Ferramenta product classification [90, 21]; social media name tagging (Twitter and Snapchat) [49]; social media target-oriented sen”
Does My Multimodal Model Learn Cross-modal Interactions? It’s Harder to Tell than You Might Think! (2020, methodology): “10For reproducibility, we used ResNet-18 features for the Kruk et al. (2019) datasets; more detail in Appendix E. 11Feature extraction approaches are known to produce competitive results relative to full fine-tuning (Devlin et al., 2019, §5.3); in some cases, mean pooling has been found to be…”
Exploring Visual Engagement Signals for Representation Learning (2021, background): “This area includes (1) detecting cyberbullying and hate speech [29, 71, 20, 40], (2) identifying emotions [43, 1, 59, 62, 81], (3) understanding rhetoric and intentions [36, 37, 70, 30, 31, 88, 74, 45, 34].”