Towards Reading Hidden Emotions: A Comparative Study of Spontaneous Micro-Expression Spotting and Recognition Methods

By Xiaobai Li, Xiaopeng Hong, Antti Moilanen, Xiaohua Huang, Tomas Pfister, Guoying Zhao, Matti Pietikäinen. IEEE Transactions on Affective Computing, 2015

392 citations recorded by the source corpus.

Micro-expressions (MEs) are rapid, involuntary facial expressions which reveal emotions that people do not intend to show. Studying MEs is valuable as recognizing them has many important applications, particularly in forensic science and psychotherapy. However, analyzing spontaneous MEs is very challenging due to their short duration and low intensity. Automatic ME analysis includes two tasks: ME spotting and ME recognition. For ME spotting, previous studies have focused on posed rather than spontaneous videos. For ME recognition, the performance of previous studies is low. To address these challenges, we make the following contributions: (i) We propose the first method for spotting spontaneous MEs in long videos (by exploiting feature difference contrast). This method is training free and works on arbitrary unseen videos. (ii) We present an advanced ME recognition framework, which outperforms previous work by a large margin on two challenging spontaneous ME databases (SMIC and CASMEII). (iii) We propose the first automatic ME analysis system (MESR), which can spot and recognize MEs from spontaneous video data. Finally, we show our method outperforms humans in the ME recognition task by a large margin, and achieves comparable performance to humans at the very challenging task of spotting and then recognizing spontaneous MEs.

Subfields: Computer Vision and Pattern Recognition, Experimental and Cognitive Psychology, Social Psychology

View Paper: https://www.semanticscholar.org/paper/29cb1e90d1e08f0990925c8d4e8d00fa5fa49100

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Experimental and Cognitive Psychology

In Social Psychology

Citations in the corpus

Survey on Deep Neural Networks in Speech and Vision Systems (2019, background): “On the other hand, visual images from videos are used to recognize human behavioral contents [137] such as facial expressions, head motion, human pose, and gestures to support a variety of applications for security, surveillance, and forensics [138-140] and”
Facial Expression Analysis under Partial Occlusion (2018, methodology): “…less attention compared to general expressions, various types of tools and systems have been developed for recognizing MEs, such as the Micro Expression Training Tool developed by Ekman [Ekman 2003], the MR recognition system [Wu et al. 2011], and the ME analysis system (MESR) [Li et al. 2017b].”
Remote Heart Rate Measurement From Highly Compressed Facial Videos: An End-to-End Deep Learning Solution With Video Enhancement (2019, background): “, chrominancebased rPPG (CHROM) [4], projection plane orthogonal to the skin tone (POS) [35], and spatial subspace rotation [36, 34, 13].”
Video-Based Facial Micro-Expression Analysis: A Survey of Datasets, Features and Algorithms (2021, methodology): “[67] made use of the Kanade-Lucas-Tomasi algorithm [104] to track three specific points of every frame.”
A Boost in Revealing Subtle Facial Expressions: A Consolidated Eulerian Framework (2019, methodology): “For fair comparison, we use the same classifier and three mainstream temporal-spatial feature descriptors: LBP-TOP6, HOG-TOP5 and HIGO-TOP7, which are extensively evaluated in the state of the art7.”