Equal But Not The Same: Understanding the Implicit Relationship Between Persuasive Images and Text
By Mingda Zhang, Rebecca Hwa, Adriana Kovashka. 2018
61 citations recorded by the source corpus.
Images and text in advertisements interact in complex, non-literal ways. The two channels are usually complementary, with each channel telling a different part of the story. Current approaches, such as image captioning methods, only examine literal, redundant relationships, where image and text show exactly the same content. To understand more complex relationships, we first collect a dataset of advertisement interpretations for whether the image and slogan in the same visual advertisement form a parallel (conveying the same message without literally saying the same thing) or non-parallel relationship, with the help of workers recruited on Amazon Mechanical Turk. We develop a variety of features that capture the creativity of images and the specificity or ambiguity of text, as well as methods that analyze the semantics within and across channels. We show that our method outperforms standard image-text alignment approaches on predicting the parallel/non-parallel relationship between image and text.
Subfields: Computer Vision and Pattern Recognition, Experimental and Cognitive Psychology, Multimodal Learning, Social Sciences & Humanities
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Douwe Kiela
- Vedanuj Goswami
- Amanpreet Singh
- Davide Testuggine
- Pratik Ringshia
- Aravind Mohan
- Hamed Firooz
- Mohit Bansal
In Experimental and Cognitive Psychology
In Multimodal Learning
- Davide Testuggine
- Pratik Ringshia
- Douwe Kiela
- Aravind Mohan
- Hamed Firooz
- Vedanuj Goswami
- Amanpreet Singh
- Ronghang Hu
In Social Sciences & Humanities
Citations in the corpus
- The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, methodology): “Other datasets include grounded entailment, which exploits the fact that one of the large-scale natural language inference datasets was constructed using captions as premises, yielding a image, premise, hypothesis triplet with associated entailment label [79]; as well as MM-IMDB, where the aim is to”