Emotion Recognition in Speech using Cross-Modal Transfer in the Wild

By Samuel Albanie, Arsha Nagrani, Andrea Vedaldi, Andrew Zisserman. 2018

291 citations recorded by the source corpus.

Obtaining large, human labelled speech datasets to train models for emotion recognition is a notoriously challenging task, hindered by annotation cost and label ambiguity. In this work, we consider the task of learning embeddings for speech classification without access to any form of labelled audio. We base our approach on a simple hypothesis: that the emotional content of speech correlates with the facial expression of the speaker. By exploiting this relationship, we show that annotations of expression can be transferred from the visual domain (faces) to the speech domain (voices) through cross-modal distillation. We make the following contributions: (i) we develop a strong teacher network for facial emotion recognition that achieves the state of the art on a standard benchmark; (ii) we use the teacher to train a student, tabula rasa, to learn representations (embeddings) for speech emotion recognition without access to labelled audio data; and (iii) we show that the speech emotion embedding can be used for speech emotion recognition on external benchmark datasets. Code, models and data are available.

Subfields: Computer Vision and Pattern Recognition, Experimental and Cognitive Psychology, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Experimental and Cognitive Psychology

In Signal Processing

Citations in the corpus

Knowledge Distillation: A Survey (2020, background): “…(Chebotar and Waters 2016; Wong and Gales 2016; Chan et al. 2015; Price et al. 2016; Fukuda et al. 2017; Bai et al. 2019; Ng et al. 2018; Albanie et al. 2018; Lu et al. 2017; Shi et al. 2019a; Roheda et al. 2018; Shi et al. 2019b; Gao et al. 2019; Ghorbani et al. 2018; Takashima et al.…”
TransFER: Learning Relation-aware Facial Expression Representations with Transformers (2021, methodology): “FERPlus PLD [3] 85.10 RAN [28] 88.55 SeNet50 [2] 88.80 RAN-VGG16 [28] 89.16 SCN [27] 89.35 KTN [17] 90.49 TransFER (Ours) 90.83 Tabel 6 compares the performance of our TransFER with the state-of-the-art methods on FERPlus.”
VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency (2021, background): “Leveraging this link, cross-modal learning methods explore a range of interesting tasks: face reconstruction from audio [54], talking face generation [82], emotion recognition [5], speaker diarization [13], speech recognition [16], and speaker identification [50, 49, 39, 74, 17].”
Cross-Modal Knowledge Distillation for Action Recognition (2019, methodology): “Cross-modal distillation has been also used for other tasks such as object detection [20], emotion recognition [21], or human pose estimation [22].”
Region Attention Networks for Pose and Occlusion Robust Facial Expression Recognition (2019, background): “Both [4] and [1] leverage the label distribution for each face as supervision.”
Learnable PINs: Cross-Modal Embeddings for Person Identity (2018, background): “ of the two modalities. While joint embeddings have been researched intensively for imLearnable PINs 3 ages and text, [5, 17, 18, 28, 49], they have also started to gain traction for audio and vision [1, 4, 37, 44]. There are several ways in which this embedding may be learnedÑwe take inspiration fr”
Voxceleb: Large-scale speaker verification in the wild (2020, background): “…et al., 2018; Ephrat et al., 2018), cross-modal transfer from face to voice or vice versa (Nagrani et al., 2018b; 2018a), emotion recognition (Albanie et al., 2018) and training face recognition from video to complement existing face recognition datasets (Cao et al., 2018;…”
More Grounded Image Captioning by Distilling Image-Text Matching Model (2020, unknown): “[15] proposed to distill the knowledge from an ensemble of models into a single model, there are a lot of follow-up works, including exploring different forms of knowledge [49, 25], cross-modality distillation [13, 1], cross-task distillation [34, 63].”
Multimodal and Temporal Perception of Audio-visual Cues for Emotion Recognition (2019, background): “However, much of the focus in AVER systems relies on multimodal learning and fusion, through the selection of apex moments [11], and there is still less exploration of multimodal interaction over time [12].”
Listen to Look: Action Recognition by Previewing Audio (2019, background): “Other forms of cross-modal distillation consider transferring supervision from RGB to flow or depth [27] or from a visual network to an audio network, or vice versa [7, 43, 3, 18].”