Automatic Understanding of Image and Video Advertisements

By Zaeem Hussain, Mingda Zhang, Xiaozhong Zhang, Keren Ye, Christopher Thomas, Zuha Agha, Nathan Ong, Adriana Kovashka. 2017

200 citations recorded by the source corpus.

There is more to images than their objective physical content: for example, advertisements are created to persuade a viewer to take a certain action. We propose the novel problem of automatic advertisement understanding. To enable research on this problem, we create two datasets: an image dataset of 64,832 image ads, and a video dataset of 3,477 ads. Our data contains rich annotations encompassing the topic and sentiment of the ads, questions and answers describing what actions the viewer is prompted to take and the reasoning that the ad presents to persuade the viewer (What should I do according to this ad, and why should I do it?), and symbolic references ads make (e.g. a dove symbolizes peace). We also analyze the most common persuasive strategies ads use, and the capabilities that computer vision systems should have to understand these strategies. We present baseline classification results for several prediction tasks, including automatically answering questions about the messages of the ads.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, methodology): “Other datasets include grounded entailment, which exploits the fact that one of the large-scale natural language inference datasets was constructed using captions as premises, yielding a image, premise, hypothesis triplet with associated entailment label [79]; as well as MM-IMDB, where the aim is to”
A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge (2022, background): “Additionally, the Ads Dataset [22] is a dataset requiring knowledge about the topic and sentiments of the ads.”
Does My Multimodal Model Learn Cross-modal Interactions? It’s Harder to Tell than You Might Think! (2020, background): “Recent computational work has examined diverse multimodal corpora collected from in-vivo social processes, e.g., visual/textual advertisements (Hussain et al., 2017; Ye and Kovashka, 2018; Zhang et al., 2018), images with non-literal captions in news articles (Weiland et al., 2018), and image/text…”
Equal But Not The Same: Understanding the Implicit Relationship Between Persuasive Images and Text (2018, methodology): “We applied it to images from the dataset of [14], and removed 11582 images that were predicted to be vintage.”
Persuasive Faces: Generating Faces in Advertisements (2018, methodology): “We focus on the Ads Dataset of [11].”
Exploring Visual Engagement Signals for Representation Learning (2021, background): “This area includes (1) detecting cyberbullying and hate speech [29, 71, 20, 40], (2) identifying emotions [43, 1, 59, 62, 81], (3) understanding rhetoric and intentions [36, 37, 70, 30, 31, 88, 74, 45, 34].”
Integrating Text and Image: Determining Multimodal Document Intent in Instagram Posts (2019, background): “Hussain et al. (2017) studies the understanding of image and video advertisements, predicting topic, sentiment, and intent.”