EmotionMeter: A Multimodal Framework for Recognizing Human Emotions

By Wei-Long Zheng, Wei Liu, Yifei Lu, Bao-Liang Lu, Andrzej S. Cichocki. IEEE Transactions on Cybernetics, 2019

1,124 citations recorded by the source corpus.

In this paper, we present a multimodal emotion recognition framework called EmotionMeter that combines brain waves and eye movements. To increase the feasibility and wearability of EmotionMeter in real-world applications, we design a six-electrode placement above the ears to collect electroencephalography (EEG) signals. We combine EEG and eye movements for integrating the internal cognitive states and external subconscious behaviors of users to improve the recognition accuracy of EmotionMeter. The experimental results demonstrate that modality fusion with multimodal deep neural networks can significantly enhance the performance compared with a single modality, and the best mean accuracy of 85.11% is achieved for four emotions (happy, sad, fear, and neutral). We explore the complementary characteristics of EEG and eye movements for their representational capacities and identify that EEG has the advantage of classifying happy emotion, whereas eye movements outperform EEG in recognizing fear emotion. To investigate the stability of EmotionMeter over time, each subject performs the experiments three times on different days. EmotionMeter obtains a mean recognition accuracy of 72.39% across sessions with the six-electrode EEG and eye movement features. These experimental results demonstrate the effectiveness of EmotionMeter within and between sessions.

Subfields: Cognitive Neuroscience, Experimental and Cognitive Psychology, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/53c36c1e5f8639457ffe6db25515e2f8ad25f61e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Experimental and Cognitive Psychology

In Human-Computer Interaction

Citations in the corpus

A Novel Bi-Hemispheric Discrepancy Model for EEG Emotion Recognition (2019, methodology): “To validate the superiority of BiHDM, we also conduct the same experiments using twelve methods, including linear support vector machine (SVM) [23], random forest (RF) [24], canonical correlation analysis (CCA) [25], group sparse canonical correlation analysis (GSCCA) [10], deep believe network (DBN”
EEG Based Emotion Recognition: A Tutorial and Review (2022, methodology): “2020 Jia et al. [153] SEED, SEED-IV a novel spatial-spectral-temporal based attention 3D dense network (SST-EmotionNet) works with 3D differential entropy representations SEED-3 classes: 96.02%, SEED-IV-4 classes: 84.92% 2020 Luo et al. [214] SEED, DEAP a method based on two deep generative models, ”
Comparing Recognition Performance and Robustness of Multimodal Deep Learning Models for Multimodal Emotion Recognition (2021, methodology): “The test schemes for emotion recognition tasks of these datasets are different in the original papers [6], [21], [45], [25], [46].”
Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCI (2024, background): “• SEED Series (Zheng & Lu, 2015; Zheng et al., 2018; Liu et al., 2022): (von Weltin et al., 2017): This is another subset of TUEG that contains annotations of slowing events (23 channels, 256 Hz).”
An Efficient LSTM Network for Emotion Recognition From Multichannel EEG Signals (2020, methodology): “[38] selected six symmetrical temporal electrode locations, excluding the frontal lobe, for”
Transfer Learning for EEG-Based Brain–Computer Interfaces: A Review of Progress Made Since 2016 (2020, positive): “workload levels from EEG. The weights of the shallow hidden neurons of the SDAE were adaptively updated during the testing phase, using augmented testing samples and their pseudo-labels. Zheng et al. [95] presented a multi-modal emotion recognition framework called EmotionMeter that combines brain w”