Dynamic Bayesian Networks for Audio-Visual Speech Recognition

By Ara V. Nefian, Luhong Liang, Xiaobo Pi, Xiaoxing Liu, Kevin P. Murphy. EURASIP Journal on Advances in Signal Processing, 2002

337 citations recorded by the source corpus.

The use of visual features in audio-visual speech recognition (AVSR) is justified by both the speech generation mechanism, which is essentially bimodal in audio and visual representation, and by the need for features that are invariant to acoustic noise perturbation. As a result, current AVSR systems demonstrate significant accuracy improvements in environments affected by acoustic noise. In this paper, we describe the use of two statistical models for audio-visual integration, the coupled HMM (CHMM) and the factorial HMM (FHMM), and compare the performance of these models with the existing models used in speaker dependent audio-visual isolated word recognition. The statistical properties of both the CHMM and FHMM allow to model the state asynchrony of the audio and visual observation sequences while preserving their natural correlation over time. In our experiments, the CHMM performs best overall, outperforming all the existing models and the FHMM.

Subfields: Signal Processing

View Paper: https://www.semanticscholar.org/paper/c4b098a76310b323e0a191ac6e6dbd0d39e15a15

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Signal Processing

Citations in the corpus

Data Fusion and Multicue Data Matching by Diffusion Maps (2006, methodology): “The combination of audiovisual cues was used in [36], where the visual cues are the extracted lip contours which are tracked over time.”
Multimodal fusion framework: A multiresolution approach for emotion classification and recognition from physiological signals (2014, positive): “Dynamic Bayesian networks Feature Nefian et al. (2002) Audio and”
Recent advances in the automatic recognition of audiovisual speech (2003, methodology): “However, with few exceptions [ 30 ], [122], proper parameter tying is usually not enforced.”
Multimodal fusion for multimedia analysis: a survey (2010, methodology): “[86] who used CHMM, Bengio [17] has presented the asynchronous HMM (AHMM) at the”
Graphical model architectures for speech recognition (2005, background): “Examples include an audio and a video feature stream [2], [21], [22], [ 39 ], [40], differing streams of audio features [48], different articulatory sequences [17], [19], [30], [33], [34], [43], or different spectral subband streams [14], [23], [35].”
DYNOTEARS: Structure Learning from Time-Series Data (2020, background): “…successfully in a variety of domains such as clinical disease prognosis (Van Gerven et al., 2008; Zandon`a et al., 2019), gene regulatory network (Linzner et al., 2019), facial and speech recognition (Meng et al., 2019; Neﬁan et al., 2002), neuroscience (Rajapakse and Zhou, 2007), among others.”
Audio-Visual Automatic Speech Recognition: An Overview (2004, background): “Notice that, in each case, matrix P can have as rows the image transform matrix rows that maximize the transformed data energy over the training set (Potamianos et al., 1998), or alternatively, that correspond to a-priori chosen locations (Nefian et al., 2002).”
Adaptive Multimodal Fusion by Uncertainty Compensation With Application to Audiovisual Speech Recognition (2009, methodology): “Combining the linearized feature degradation model of (24) with the clean speech GMM yields the improved enhanced audio feature estimate (25) where is the assignment probability of the audio feature to the th clean speech GMM mixture component after the th iteration of the enhancement process.”
A survey of sentiment analysis in social media (2018, background): “Besides, there are novel fusion types such as hybrid multimodal fusion combining both of feature-level fusion and decision-level fusion methods [97, 99], model-level fusion using the correlation between data under different modalities [9, 142], rule-based fusion [28], classification-based fusion usi”