Seeing Voices and Hearing Faces: Cross-Modal Biometric Matching

By Arsha Nagrani, Samuel Albanie, Andrew Zisserman. 2018

253 citations recorded by the source corpus.

We introduce a seemingly impossible task: given only an audio clip of someone speaking, decide which of two face images is the speaker. In this paper we study this, and a number of related cross-modal tasks, aimed at answering the question: how much can we infer from the voice about the face and vice versa? We study this task "in the wild", employing the datasets that are now publicly available for face recognition from static images (VGGFace) and speaker identification from audio (VoxCeleb). These provide training and testing scenarios for both static and dynamic testing of cross-modal matching. We make the following contributions: (i) we introduce CNN architectures for both binary and multi-way cross-modal face and audio matching: (ii) we compare dynamic testing (where video information is available, but the audio is not from the same video) with static testing (where only a single still image is available): and (iii) we use human testing as a baseline to calibrate the difficulty of the task. We show that a CNN can indeed be trained to solve this task in both the static and dynamic scenarios, and is even well above chance on 10-way classification of the face given the voice. The CNN matches human performance on easy examples (e.g. different gender across faces) but exceeds human performance on more challenging examples (e.g. faces with the same gender, age and nationality).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

VoxCeleb2: Deep Speaker Recognition (2018, background): “…for example – visual speech synthesis [9, 10], speech separation [11, 12], † These authors contributed equally to this work. cross-modal transfer from face to voice or vice versa [13, 14] and training face recognition from video to complement existing face recognition datasets [4, 5, 6].”
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, background): “This idea has been further developed for visually-guided audio source separation [43–48], sound generation from silent videos [39, 49], sound localization in video frames [50, 51], face and voice association [52, 53], and video-based audio localization [54, 55].”
The Sound of Pixels (2018, background): “Sound source separation methods enable applications ranging from music/vocal separation [24], to speech separation and enhancement [25,26,27].”
Deep Audio-visual Learning: A Survey (2020, methodology): “Nagrani et al. [39] The method is novel and incorporates dynamic information As the sample size increases, the accuracy decreases excessively Wen et al. [42] .”
Audiovisual SlowFast Networks for Video Recognition (2020, background): “Other audiovisual tasks include audio-visual speech recognition [60, 56], lip reading [12], biometric matching [55], sound-source localization [10, 3, 65, 86, 34, 15], audiovisual source separation [58, 23], and audiovisual question answering [1].”
Grounding Spoken Words in Unlabeled Video (2019, background): “This idea has been further developed for visually-guided audio source separation [8,18,19], generating sounds from silent videos [17,21], localizing sounds in video frames [3, 19], learning association of faces and voices [13,15], and video-based audio spatialization [7,14].”
VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency (2021, background): “association, supported by cognitive science studies [11], is today often leveraged for speaker identification given the recording of a single speaker [50, 49, 39, 74].”
Deep Multimodal Feature Encoding for Video Ordering (2020, unknown): “Audio-visual representation learning has seen applications in several tasks such as event classification [40], audiovisual localization [40, 2], biometric matching [35], sound localization [37, 2], person identification [34], action recognition [37], on/off-screen audio separation [12, 37], and vide”
Learnable PINs: Cross-Modal Embeddings for Person Identity (2018, methodology): “In our earlier work [28] we established, by using a forced matching task, that strong correlations exist between faces and voices belonging to the same identity.”
Emotion Recognition in Speech using Cross-Modal Transfer in the Wild (2018, background): “h (an idea that was previously explored in the neuroscience community [9]). Some works have also examined cross-modal relationships between faces and voices in order to learn identity representations [39, 48, 49]. Differently from these works, our approach places an explicit reliance on the correspo”
Voxceleb: Large-scale speaker verification in the wild (2020, background): “…2017; Karras et al., 2017), speech separation (Afouras et al., 2018; Ephrat et al., 2018), cross-modal transfer from face to voice or vice versa (Nagrani et al., 2018b; 2018a), emotion recognition (Albanie et al., 2018) and training face recognition from video to complement existing face…”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, background): “Following this, many novel crossmodal KD methods [136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 152, 153, 156] have been proposed.”