Audio-Visual Speech Modeling for Continuous Speech Recognition

By Stéphane Dupont, Juergen Luettin. IEEE Trans. Multim., 2000

637 citations recorded by the source corpus.

This paper describes a speech recognition system that uses both acoustic and visual speech information to improve recognition performance in noisy environments. The system consists of three components: a visual module; an acoustic module; and a sensor fusion module. The visual module locates and tracks the lip movements of a given speaker and extracts relevant speech features. This task is performed with an appearance-based lip model that is learned from example images. Visual speech features are represented by contour information of the lips and grey-level information of the mouth area. The acoustic module extracts noise-robust features from the audio signal. Finally the sensor fusion module is responsible for the joint temporal modeling of the acoustic and visual feature streams and is realized using multistream hidden Markov models (HMMs). The multistream method allows the definition of different temporal topologies and levels of stream integration and hence enables the modeling of temporal dependencies more accurately than traditional approaches. We present two different methods to learn the asynchrony between the two modalities and how to incorporate them in the multistream models. The superior performance for the proposed system is demonstrated on a large multispeaker database of continuously spoken digits. On a recognition task at 15 dB acoustic signal-to-noise ratio (SNR), acoustic perceptual linear prediction (PLP) features lead to 56% error rate, noise robust RASTA-PLP (relative spectra) acoustic features to 7.2% error rate and combined noise robust acoustic features and visual features to 2.5% error rate.

Subfields: Artificial Intelligence, Signal Processing

View Paper: https://www.semanticscholar.org/paper/8943a6703948d71eeb48ce20b1ef94799233d757

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Lipreading Using Temporal Convolutional Networks (2020, methodology): “Traditionally, a two-stage approach was followed where features were first extracted from the mouth region, with the Discrete Cosine Transform being the most popular feature extractor, and then fed to a Hidden Markov Model (HMMs) for modeling of the temporal dynamics [1, 2, 3].”
Automatic analysis of multimodal group actions in meetings (2005, methodology): “Multistreammodels are typically employedwith separate streams for audio and visual features in multimodal tasks [21], or for different frequency subbands in speech recognition [20].”
End-To-End Audio-Visual Speech Recognition with Conformers (2021, methodology): “Traditional audio-visual speech recognition methods follow a two-step approach, feature extraction and recognition [9, 26].”
End-to-End Audiovisual Speech Recognition (2018, positive): “Traditional audiovisual fusion systems consist of two stages, feature extraction from the image and audio signals and combination of the features for joint classification [1, 2, 3].”
Recent advances in the automatic recognition of audiovisual speech (2003, methodology): “product (or, coupled) HMMs have been considered in some small-vocabulary recognition tasks [26], [28], [29], [70], [80], [105], where synchronization is sometimes enforced at the word level, and recently for LVCSR [37], [104].”
Audio-Visual Automatic Speech Recognition: An Overview (2004, methodology): “…of the two classifiers into a joint audio-visual classification score, using appropriate weights that capture the reliability of each single-modality classifier, or data stream (Hennecke et al., 1996; Rogozan et al., 1997; Potamianos and Graf, 1998; Dupont and Luettin, 2000; Neti et al., 2000).”
Deep Audio-visual Learning: A Survey (2020, methodology): “…is not considered Afouras et al. [61] Study noise in audio LRS2-BBC Dataset Complex network Earlier efforts on audio-visual fusion models usually consisted of two steps: 1) extracting features from the image and audio signals and 2) combining the features for joint classiﬁcation [62, 63, 64] .”
Dynamic Bayesian Networks for Audio-Visual Speech Recognition (2002, methodology): “The audio-visual multistream product HMM [11] [14] [15] [16] [17] illustrated in Figure 3, can be seen as an extension of the previous model by representing each hidden state of the multistream HMM as a pair of one audio and one visual state.”
Multimodal Intelligence: Representation Learning, Information Fusion, and Applications (2019, background): “vision can also be combined with speech to perform audio-visual speech recognition [53]–[55], speaker recognition [56]–[58],”
Weakly Supervised Learning with Multi-Stream CNN-LSTM-HMMs to Discover Sequential Parallelism in Sign Language Videos (2020, background): “nition [8], [9], audio-visual speech recognition [10], [11], action recognition [12] and sign language recognition [13], [14], [15], [16].”
Audio-Visual Speech Recognition Using Lip Information Extracted from Side-Face Images (2007, background): “Audio-visual (bimodal) speech recognition techniques using face information in addition to acoustic information are promising directions for increasing the robustness of speech recognition, and many audio-visual methods have been proposed thus far [1–11].”
Adaptive Multimodal Fusion by Uncertainty Compensation With Application to Audiovisual Speech Recognition (2009, background): “A challenging application domain for multimodal fusion schemes is audiovisual automatic speech recognition (AV-ASR), since it requires modeling both the relative reliability and the synchronicity of the audio and visual modalities.”
Automatic nonverbal analysis of social interaction in small groups: A review (2009, methodology): “A number of HMM variations were tested, including multistream HMMs [43], coupled HMMs [14], and asynchronous HMMs [12].”