Robust Image Sentiment Analysis Using Progressively Trained and Domain Transferred Deep Networks

By Quanzeng You, Jiebo Luo, Hailin Jin, Jianchao Yang. AAAI Conference on Artificial Intelligence, 2015

580 citations recorded by the source corpus.

Sentiment analysis of online user generated content is important for many social media analytics tasks. Researchers have largely relied on textual sentiment analysis to develop systems to predict political elections, measure economic indicators, and so on. Recently, social media users are increasingly using images and videos to express their opinions and share their experiences. Sentiment analysis of such large scale visual content can help better extract user sentiments toward events or topics, such as those in image tweets, so that prediction of sentiment from visual content is complementary to textual sentiment analysis. Motivated by the needs in leveraging large scale yet noisy training data to solve the extremely challenging problem of image sentiment analysis, we employ Convolutional Neural Networks (CNN). We first design a suitable CNN architecture for image sentiment analysis. We obtain half a million training samples by using a baseline sentiment algorithm to label Flickr images. To make use of such noisy machine labeled data, we employ a progressive strategy to fine-tune the deep network. Furthermore, we improve the performance on Twitter images by inducing domain transfer with a small number of manually labeled Twitter images. We have conducted extensive experiments on manually labeled Twitter images. The results show that the proposed CNN can achieve better performance in image sentiment analysis than competing algorithms.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/39e4636be6eb9903151847c0665503133971d63c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Building a Large Scale Dataset for Image Emotion Recognition: The Fine Print and The Benchmark (2016, methodology): “2012) to fine-tune another model as described in (You et al. 2015).”
Multimodal Sentiment Analysis: A Survey of Methods, Trends, and Challenges (2023, methodology): “[96] Twitter and Flickr Dataset Design framework using a CNN architecture and progressive strategy to fine-tune the deep network for image sentiment analysis 87.”
A survey of multimodal sentiment analysis (2017, methodology): “[125,126] explored fine-tuning in combination with SVMs and logistic regression classifiers, rather than disjointly, for visual sentiment analysis on the same set of Twitter images as in [124] with 6.”
Privacy-CNH: A Framework to Detect Photo Privacy with Convolutional Neural Network using Hierarchical Features (2016, methodology): “SVM with BOVW: The work of (Zerr et al. 2012) and (Squicciarini, Caragea, and Balakavi 2014) both explored SIFT descriptors by (Lowe 2004) to extract features from images and learned a visual word dictionary.”
Sentiment analysis using deep learning architectures: a review (2019, background): “• Twitter image dataset (You et al. 2015) consists of 1269 image tweets labeled by five Amazon Mechanical Turk (AMT) workers.”
How Polarized Have We Become? A Multimodal Classification of Trump Followers and Clinton Followers (2017, background): “Individual attributes such as age, gender, sentiment and even income, can be estimated using either texts [2, 39, 26] or images [17, 42].”
Deep learning for image classification on very small datasets using transfer learning (2019, background): “In computer vision, examples of transfer learning including [16] [17] lead to signiﬁcantly improved results for object classiﬁcation.”