A review of affective computing: From unimodal analysis to multimodal fusion

By Soujanya Poria, Erik Cambria, Rajiv Bajpai, Amir Hussain. 2017

1,555 citations recorded by the source corpus.

First review on affective computing that is dealing with both unimodal and multimodal analysis.The survey takes into account recent approaches, e.g., embeddings, which are missing from previous reviews.It covers and compares all state-of-the-art methods in details, while most available surveys just quickly describes them. Affective computing is an emerging interdisciplinary research field bringing together researchers and practitioners from various fields, ranging from artificial intelligence, natural language processing, to cognitive and social sciences. With the proliferation of videos posted online (e.g., on YouTube, Facebook, Twitter) for product reviews, movie reviews, political views, and more, affective computing research has increasingly evolved from conventional unimodal analysis to more complex forms of multimodal analysis. This is the primary motivation behind our first of its kind, comprehensive literature review of the diverse field of affective computing. Furthermore, existing literature surveys lack a detailed discussion of state of the art in multimodal affect analysis frameworks, which this review aims to address. Multimodality is defined by the presence of more than one modality or channel, e.g., visual, audio, text, gestures, and eye gage. In this paper, we focus mainly on the use of audio, visual and text information for multimodal affect analysis, since around 90% of the relevant literature appears to cover these three modalities. Following an overview of different techniques for unimodal affect analysis, we outline existing methods for fusing information from different modalities. As part of this review, we carry out an extensive study of different categories of state-of-the-art fusion techniques, followed by a critical analysis of potential performance improvements with multimodal analysis compared to unimodal analysis. A comprehensive overview of these two complementary fields aims to form the building blocks for readers, to better understand this challenging and exciting research field.

Subfields: Artificial Intelligence, Experimental and Cognitive Psychology, Social Psychology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Experimental and Cognitive Psychology

In Social Psychology

Citations in the corpus

ICON: Interactive Conversational Memory Network for Multimodal Emotion Detection (2018, methodology): “Thus, in our experiments, we extract video-level contextual features for utterances from each modality using the network proposed by Poria et al. 2017b.”
Tensor Fusion Network for Multimodal Sentiment Analysis (2017, background): “…sentiment analysis (Morency et al., 2011; Zadeh et al., 2016b; Poria et al., 2015) is an increasingly popular area of affective computing research (Poria et al., 2017) that focuses on generalizing text-based sentiment analysis to opin-ionated videos, where three communicative modalities are…”
Context-Dependent Sentiment Analysis in User-Generated Videos (2017, methodology): “Poria et al. (Poria et al., 2015, 2016d, 2017b) extracted audio, visual and textual features using convolutional neural network (CNN); concatenated those features and employed multiple kernel learning (MKL) for final sentiment classification.”
Multimodal sentiment analysis: A systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directions (2022, background): “The early studies [87] , [88] and [89] show that employing any combination of three modalities, a bimodal system and trimodal system have superior accuracy than a unimodal system.”
Comparing Recognition Performance and Robustness of Multimodal Deep Learning Models for Multimodal Emotion Recognition (2021, methodology): “We select the output dimension from the set [5, 10, 15, 20, 25, 30] and the EEG weight α1 in [0, 0.”
IoT-Enabled Social Relationships Meet Artificial Social Intelligence (2021, background): “[37], the field that encompasses the development and design of devices and systems that can capture, process and detect human effects, and Personality Computing [38], the study field that aim to integrate human personality traits in the computing systems, and many other social-related computing fiel”
Multimodal sentiment analysis with word-level fusion and reinforcement learning (2017, methodology): “Multimodal models have been used for sentiment analysis [28, 45, 46]; medical purposes, such as detection of suicidal risk, PTSD and depression [29, 34, 35, 43]; emotion recognition [23]; image captioning and media description [8, 41]; question answering [2]; and multimodal translation [31].”
Natural language based financial forecasting: a survey (2017, background): “Existing approaches to affective computing fall into three categories: knowledge-based techniques, statistical methods, and hybrid approaches [91].”
Radical-Based Hierarchical Embeddings for Chinese Sentiment Analysis at Sentence Level (2017, background): “It is only when they are placed in a certain order that their connection with sentiment begins to reveal (Poria et al. 2017).”
Continual Learning for Affective Robotics: Why, What and How? (2020, background): “Although the current (deep) learning-based approaches provide high performance on affect recognition and classification benchmarks (see [12]–[14] for an overview), they are not able to translate this performance to real-world situations where robots need to dynamically interact with different users.”