Self-Supervised ECG Representation Learning for Emotion Recognition

By Pritam Sarkar, Ali Etemad. IEEE Transactions on Affective Computing, 2020

355 citations recorded by the source corpus.

We exploit a self-supervised deep multi-task learning framework for electrocardiogram (ECG) -based emotion recognition. The proposed solution consists of two stages of learning a) learning ECG representations and b) learning to classify emotions. ECG representations are learned by a signal transformation recognition network. The network learns high-level abstract representations from unlabeled ECG data. Six different signal transformations are applied to the ECG signals, and transformation recognition is performed as pretext tasks. Training the model on pretext tasks helps the network learn spatiotemporal representations that generalize well across different datasets and different emotion categories. We transfer the weights of the self-supervised network to an emotion recognition network, where the convolutional layers are kept frozen and the dense layers are trained with labelled ECG data. We show that the proposed solution considerably improves the performance compared to a network trained using fully-supervised learning. New state-of-the-art results are set in classification of arousal, valence, affective states, and stress for the four utilized datasets. Extensive experiments are performed, providing interesting insights into the impact of using a multi-task self-supervised structure instead of a single-task model, as well as the optimum level of difficulty required for the pretext self-supervised tasks.

Subfields: Cardiology and Cardiovascular Medicine, Cognitive Neuroscience, Experimental and Cognitive Psychology

View Paper: https://www.semanticscholar.org/paper/a484fc24b62758b5bc61e6b967e7935c1a08dcab

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cardiology and Cardiovascular Medicine

In Cognitive Neuroscience

In Experimental and Cognitive Psychology

Citations in the corpus

Emotion recognition and artificial intelligence: A systematic review (2014-2023) and research recommendations (2023, neutral): “Thus, the summary of Table A.3 [53,73,74,79–83].”
ECG Monitoring Systems: Review, Architecture, Processes, and Key Challenges (2020, background): “ECG signal can be used to evaluate psychological state, emotions, and stress levels to aid mental healthcare for people living in stressful environments [59,269,270].”
CLOCS: Contrastive Learning of Cardiac Signals (2020, methodology): “), via supervised pre-training ( Supervised ), or via a multi-task pre-training mechanism introduced speciﬁcally for ECG signals ( MT-SSL ) (Sarkar & Etemad, 2020).”