Learnable PINs: Cross-Modal Embeddings for Person Identity

By Arsha Nagrani, Samuel Albanie, Andrew Zisserman. 2018

169 citations recorded by the source corpus.

We propose and investigate an identity sensitive joint embedding of face and voice. Such an embedding enables cross-modal retrieval from voice to face and from face to voice. We make the following four contributions: first, we show that the embedding can be learnt from videos of talking faces, without requiring any identity labels, using a form of cross-modal self-supervision; second, we develop a curriculum learning schedule for hard negative mining targeted to this task, that is essential for learning to proceed successfully; third, we demonstrate and evaluate cross-modal retrieval for identities unseen and unheard during training over a number of scenarios and establish a benchmark for this novel task; finally, we show an application of using the joint embedding for automatically retrieving and labelling characters in TV dramas.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

VoxCeleb2: Deep Speaker Recognition (2018, background): “…for example – visual speech synthesis [9, 10], speech separation [11, 12], † These authors contributed equally to this work. cross-modal transfer from face to voice or vice versa [13, 14] and training face recognition from video to complement existing face recognition datasets [4, 5, 6].”
Use What You Have: Video retrieval using representations from collaborative experts (2019, background): “More recently, several works have also focused on audio-visual cross-modal embeddings [2, 37], as well as audio-text embeddings [7].”
Sound-Guided Semantic Image Manipulation (2021, background): “Audio-visual representation learning studies [3, 28, 44] aim to map both modalities to the same embedding space.”
VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency (2021, methodology): “association, supported by cognitive science studies [11], is today often leveraged for speaker identification given the recording of a single speaker [50, 49, 39, 74].”
Deep Audio-visual Learning: A Survey (2020, methodology): “To solve this problem, Arsha et al. [54] proposed a cross-modality self-supervised method to learn the embedding of audio and visual information from a video and signiﬁcantly reduced the complexity of the network.”
Deep Multimodal Feature Encoding for Video Ordering (2020, unknown): “Audio-visual representation learning has seen applications in several tasks such as event classification [40], audiovisual localization [40, 2], biometric matching [35], sound localization [37, 2], person identification [34], action recognition [37], on/off-screen audio separation [12, 37], and vide”
EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition (2019, background): “These approaches attempt fusion [2,28] or embedding into a common space [3, 6, 26].”
Emotion Recognition in Speech using Cross-Modal Transfer in the Wild (2018, methodology): “Some works have also examined cross-modal relationships between faces and voices in order to learn identity representations [39, 48, 49].”
Self-supervised learning of a facial attribute embedding from video (2018, methodology): “Similarly to [37], using progressively more difﬁcult samples proved crucial for the strategy’s success.”