CREMA-D: Crowd-sourced Emotional Multimodal Actors Dataset

By Houwei Cao, David G. Cooper, M. K. Keutmann, Ruben C. Gur, Ani Nenkova, R. Verma. 2014

1,107 citations recorded by the source corpus.

People convey their emotional state in their face and voice. We present an audio-visual dataset uniquely suited for the study of multi-modal emotion expression and perception. The dataset consists of facial and vocal emotional expressions in sentences spoken in a range of basic emotional states (happy, sad, anger, fear, disgust, and neutral). 7,442 clips of 91 actors with diverse ethnic backgrounds were rated by multiple raters in three modalities: audio, visual, and audio-visual. Categorical emotion labels and real-value intensity values for the perceived emotion were collected using crowd-sourcing from 2,443 raters. The human recognition of intended emotion for the audio-only, visual-only, and audio-visual data are 40.9, 58.2 and 63.6 percent respectively. Recognition rates are highest for neutral, followed by happy, anger, disgust, fear, and sad. Average intensity levels of emotion are rated highest for visual-only perception. The accurate recognition of disgust and fear requires simultaneous audio-visual cues, while anger and happiness can be well recognized based on evidence from a single modality. The large dataset we introduce can be used to probe other questions concerning the audio-visual perception of emotion.

Subfields: Cognitive Neuroscience, Experimental and Cognitive Psychology, Evaluation & Benchmarks, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

1. Kun Zhou
2. Lele Chen
3. Haizhou Li
4. Timo Bolkart
5. Chenliang Xu
6. Zhaoxi Mu
7. Radek Daněček
8. Michael J. Black

In Experimental and Cognitive Psychology

1. Reza Lotfian
2. Carlos Busso
3. Rui Liu
4. Emily Mower Provost
5. Wei Han
6. William Chan
7. Kun Zhou
8. Lele Chen

In Evaluation & Benchmarks

1. Yu Zhang
2. Wei Han
3. Chung-Cheng Chiu
4. Yonghui Wu
5. Haizhou Li
6. Bo Li
7. William Chan
8. Kun Zhou

In Speech & Audio

1. Yu Zhang
2. Jinyu Li
3. Tara N. Sainath
4. Yonghui Wu
5. Chung-Cheng Chiu
6. Ruoming Pang
7. Bo Li
8. Wei Han

Citations in the corpus

• BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition (2021, background): “The benchmark includes a variety of tasks such as speech emotion recognition [85], [86], speaker identification [87], and language identification [88], but specifically excludes tasks that are focused on the meaning of words.”
• The Emotional Voices Database: Towards Controlling the Emotion Dimension in Voice Generation Systems (2018, methodology): “The CREMA-D database [6] is similar to the RAVDESS.”
• Realistic Speech-Driven Facial Animation with GANs (2019, neutral): “com/view/facial-animation [17], CREMA-D [6] and LRW [11] datasets.”
• Review of end-to-end speech synthesis technology based on deep learning (2021, background): “None of the abovementioned corpora contains explicit emotional information, and most of the existing emotional corpora cannot be effectively used to train the emotional TTS model based on deep learning, because these data sets contain a small number of sentences, such as RAVDESS [128], CREMA-D [23],”
• Building Naturalistic Emotionally Balanced Speech Corpus by Retrieving Emotional Speech from Existing Podcast Recordings (2019, methodology): “This approach was used in the Emo-DB, CREMA-D, and Chen Bimodal database.”
• FRILL: A Non-Semantic Speech Embedding for Mobile Devices (2020, neutral): “NOSS includes a diverse set tasks such as emotion recognition [5, 6], speaker identification [7], language identification [8], and keyword detection [9], and is designed to encourage the development of non-semantic speech embeddings.”
• Emotional Voice Conversion: Theory, Databases and ESD (2021, methodology): “Similar to the RAVDESS database, the CREMA-D database (Cao et al., 2014) contains 12 different sentences recorded by 91 actors.”
• EMOCA: Emotion Driven Monocular Face Capture and Animation (2022, background): “2 [13] Houwei Cao, David Cooper, Michael Keutmann, Ruben Gur, Ani Nenkova, and Ragini Verma.”
• Multimodal and Temporal Perception of Audio-visual Cues for Emotion Recognition (2019, methodology): “For example, in CREMA-D, the most common misleading happens between fear and sadness, with 19%.”
• MSP-IMPROV: An Acted Corpus of Dyadic Interactions to Study Emotion Perception (2017, methodology): “Second, using crowdsourcing significantly reduces the time required for the evaluation [54].”
• LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT (2023, methodology): “For the SER task, we collect corpora including MELD (Poria et al., 2018), IEMOCAP (Busso et al., 2008), RAVDESS (Livingstone & Russo, 2018), TESS (Pichora-Fuller & Dupuis, 2020), Crema-D (Cao et al., 2014), Emov-DB (Adigwe et al., 2018), and SAVEE (Jackson & Haq, 2014).”
• Balanced Multimodal Learning via On-the-fly Gradient Modulation (2022, background): “CREMA-D [5] is an audio-visual dataset for speech emotion recognition, containing 7,442 video clips of 2-3 seconds from 91 actors speaking several short words.”
• What comprises a good talking-head video generation?: A Survey and Benchmark (2020, background): “[2], the human recognition of intended emotion on the CREMA-D dataset are 58.”