Audio-Visual Speaker Diarization Based on Spatiotemporal Bayesian Fusion

By Israel Dejene Gebru, Silèye O. Ba, Xiaofei Li, Radu Patrice Horaud. 2016

126 citations recorded by the source corpus.

Speaker diarization consists of assigning speech signals to people engaged in a dialogue. An audio-visual spatiotemporal diarization model is proposed. The model is well suited for challenging scenarios that consist of several participants engaged in multi-party interaction while they move around and turn their heads towards the other participants rather than facing the cameras and the microphones. Multiple-person visual tracking is combined with multiple speech-source localization in order to tackle the speech-to-person association problem. The latter is solved within a novel audio-visual fusion method on the following grounds: binaural spectral features are first extracted from a microphone pair, then a supervised audio-visual alignment technique maps these features onto an image, and finally a semi-supervised clustering method assigns binaural spectral features to visible persons. The main advantage of this method over previous work is that it processes in a principled way speech signals uttered simultaneously by multiple persons. The diarization itself is cast into a latent-variable temporal graphical model that infers speaker identities and speech turns, based on the output of an audio-visual association process, executed at each time slice, and on the dynamics of the diarization variable itself. The proposed formulation yields an efficient exact inference procedure. A novel dataset, that contains audio-visual training data as well as a number of scenarios involving several participants engaged in formal and informal dialogue, is introduced. The proposed method is thoroughly tested and benchmarked with respect to several state-of-the art diarization algorithms.

Subfields: Computational Mechanics, Multimodal Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Multimodal Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “Datasets like VoxCeleb [39], AVA Speech [31], AVA active speaker [192], AVDIAR [83], and EasyCom [53] support this research.”
Multimodal Intelligence: Representation Learning, Information Fusion, and Applications (2019, background): “speaker diarization, [59], [60], as well as speech separation [61], [62] and enhancement [63].”
Multimodal Machine Learning: A Survey and Taxonomy (2017, methodology): “[68] trained a Gaussian mixture model and performed semi-supervised clustering together with an unsupervised latent-variable graphical model to align speakers in an audio channel with their locations in a video.”
Ava Active Speaker: An Audio-Visual Dataset for Active Speaker Detection (2019, background): “The UT-CRSS-4EnglishAccent corpus for voice activity detection [46] and the AVDIAR corpus [20] for diarization are recent corpora, but are limited in subject diversity and recording conditions.”
Self-Supervised Learning of Audio-Visual Objects from Video (2020, background): “The first one is in extending our object embeddings to other audio-visual speaker tasks, such as diarizing conversations [11, 28], and face/head detection.”