Spatial–Temporal Recurrent Neural Network for Emotion Recognition

By T. Zhang, Wenming Zheng, Zhen Cui, Yuan Zong, Yang Li. IEEE Transactions on Cybernetics, 2017

504 citations recorded by the source corpus.

In this paper, we propose a novel deep learning framework, called spatial–temporal recurrent neural network (STRNN), to integrate the feature learning from both spatial and temporal information of signal sources into a unified spatial–temporal dependency model. In STRNN, to capture those spatially co-occurrent variations of human emotions, a multidirectional recurrent neural network (RNN) layer is employed to capture long-range contextual cues by traversing the spatial regions of each temporal slice along different directions. Then a bi-directional temporal RNN layer is further used to learn the discriminative features characterizing the temporal dependencies of the sequences, where sequences are produced from the spatial RNN layer. To further select those salient regions with more discriminative ability for emotion recognition, we impose sparse projection onto those hidden states of spatial and temporal domains to improve the model discriminant ability. Consequently, the proposed two-layer RNN model provides an effective way to make use of both spatial and temporal dependencies of the input signals for emotion recognition. Experimental results on the public emotion datasets of electroencephalogram and facial expression demonstrate the proposed STRNN method is more competitive over those state-of-the-art methods.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Experimental and Cognitive Psychology

View Paper: https://www.semanticscholar.org/paper/a5b2fb8711d5771bcb078a1e21430dc5a642095a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

1. Dongrui Wu
2. Chunyan Miao
3. Di Wang
4. Peixiang Zhong
5. Chin-Teng Lin
6. Peng Xu
7. Alireza Jolfaei
8. Xiaotong Gu

In Computer Vision and Pattern Recognition

1. Yi Ding
2. Tengfei Song
3. Bin Hu
4. Tzyy-Ping Jung
5. Zhen Cui
6. Dongrui Wu
7. Xiaojun Chang
8. Cuntai Guan

In Experimental and Cognitive Psychology

1. Chunyan Miao
2. Di Wang
3. Peixiang Zhong
4. Dongrui Wu
5. Bifta Sama Bari
6. Tengfei Song
7. Bin Hu
8. Sabira Khatun

Citations in the corpus

• EEG-Based Brain-Computer Interfaces (BCIs): A Survey of Recent Studies on Signal Sensing Technologies and Computational Intelligence Approaches and Their Applications (2020, methodology): “The RNN framework has also been applied to other EEG-based tasks, such as identification of individuals [26], hand motion identification [131], sleep staging [132], and emotion recognition [133].”
• TSception: Capturing Temporal Dynamics and Spatial Asymmetry From EEG for Emotion Recognition (2021, methodology): “Different from the methods using manually extracted features [21], [22], [25], [33], [34], EEG signals are fed into TSception directly, which makes it an end-to-end deep learning method that needs less domain knowledge about the features.”
• EEG Based Emotion Recognition: A Tutorial and Review (2022, methodology): “[159] SEED a Riemannian fusion network to learn the most discriminative and complementary spatial and temporal information 3 classes (subject-dependent modeling: 93.”
• Deep learning-based electroencephalography analysis: a systematic review (2019, methodology): “Recurrent models were also applied in cases where it was necessary to capture long-term dependencies from the EEG data [111, 239].”
• EEG-Based Emotion Recognition Using Regularized Graph Neural Networks (2019, methodology): “Recently, Zhang et al. [10] and Li et al. [38] proposed to use RNNs to learn spatial topological relations between channels by scanning electrodes in both vertical and horizontal directions.”
• Making Sense of Spatio-Temporal Preserving Representations for EEG-Based Human Intention Recognition (2020, background): “Recent research has also found its applications for healthy users, such as emotion recognition [4]–[6].”
• Current Status, Challenges, and Possible Solutions of EEG-Based Brain-Computer Interface: A Comprehensive Review (2020, methodology): “Two notable studies have been carried out to recognize the EEG-based sleep stage using RNN architecture (Michielli et al., 2019; Wang and Wu, 2019; Zhang T. et al., 2019).”
• A Novel Bi-Hemispheric Discrepancy Model for EEG Emotion Recognition (2019, methodology): “[25] employed a multidirectional RNN layer to capture long-range contextual cues by traversing the spatial regions of each temporal slice along different”