Multilingual Visual Sentiment Concept Matching
By Nikolaos Pappas, Miriam Redi, Mercan Topkara, Brendan Jou, Tao Chen, Shih-Fu Chang. 2016
15 citations recorded by the source corpus.
The impact of culture in visual emotion perception has recently captured the attention of multimedia research. In this study, we provide powerful computational linguistics tools to explore, retrieve and browse a dataset of 16K multilingual affective visual concepts and 7.3M Flickr images. First, we design an effective crowdsourcing experiment to collect human judgements of sentiment connected to the visual concepts. We then use word embeddings to represent these concepts in a low dimensional vector space, allowing us to expand the meaning around concepts, and thus enabling insight about commonalities and differences among different languages. We compare a variety of concept representations through a novel evaluation task based on the notion of visual semantic relatedness. Based on these representations, we design clustering schemes to group multilingual visual concepts, and evaluate them with novel metrics based on the crowdsourced sentiment annotations as well as visual semantic relatedness. The proposed clustering framework enables us to analyze the full multilingual dataset in-depth and also show an application on a facial data subset, exploring cultural insights of portrait-related affective visual concepts.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Sentiment & Opinion Analysis
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Maja Pantic
- Mohammad Soleymani
- Björn Schuller
- Brendan Jou
- David García
- Shih-Fu Chang
- Florian Eyben
- Michel François Valstar
In Computer Vision and Pattern Recognition
- Maja Pantic
- Björn Schuller
- Mohammad Soleymani
- Shih-Fu Chang
- Brendan Jou
- David García
- Luc Van Gool
- Xiaogang Wang
In Multimodal Learning
In Sentiment & Opinion Analysis
Citations in the corpus
- A survey of multimodal sentiment analysis (2017, methodology): “In addition, they also explored generating ontology structures automatically by either grouping ANP by exact translations with a pivot language or approximate ANP groupings using word embeddings [23,117].”