Audio-visual speech recognition using deep learning

By Kuniaki Noda, Yuki Yamaguchi, Kazuhiro Nakadai, Hiroshi G. Okuno, Tetsuya Ogata. 2014

551 citations recorded by the source corpus.

Audio-visual speech recognition (AVSR) system is thought to be one of the most promising solutions for reliable speech recognition, particularly when the audio is corrupted by noise. However, cautious selection of sensory features is crucial for attaining high recognition performance. In the machine-learning community, deep learning approaches have recently attracted increasing attention because deep neural networks can effectively extract robust latent features that enable various recognition algorithms to demonstrate revolutionary generalization capabilities under diverse application conditions. This study introduces a connectionist-hidden Markov model (HMM) system for noise-robust AVSR. First, a deep denoising autoencoder is utilized for acquiring noise-robust audio features. By preparing the training data for the network with pairs of consecutive multiple steps of deteriorated audio features and the corresponding clean features, the network is trained to output denoised audio features from the corresponding features deteriorated by noise. Second, a convolutional neural network (CNN) is utilized to extract visual features from raw mouth area images. By preparing the training data for the CNN as pairs of raw images and the corresponding phoneme label outputs, the network is trained to predict phoneme labels from the corresponding mouth area input images. Finally, a multi-stream HMM (MSHMM) is applied for integrating the acquired audio and visual HMMs independently trained with the respective features. By comparing the cases when normal and denoised mel-frequency cepstral coefficients (MFCCs) are utilized as audio features to the HMM, our unimodal isolated word recognition results demonstrate that approximately 65 % word recognition rate gain is attained with denoised MFCCs under 10 dB signal-to-noise-ratio (SNR) for the audio signal input. Moreover, our multimodal isolated word recognition results utilizing MSHMM with denoised MFCCs and acquired visual features demonstrate that an additional word recognition rate gain is attained for the SNR conditions below 10 dB.

Subfields: Artificial Intelligence, Natural Language Processing, Multimodal Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Multimodal Learning

In Signal Processing

Citations in the corpus

Local Aggregation for Unsupervised Learning of Visual Embeddings (2019, background): “Nevertheless, unsupervised networks are still significantly lower performing than their supervised counterparts, and are rarely used in real-world applications [7, 51, 8].”
Deep Learning Approach for Short-Term Stock Trends Prediction Based on Two-Stream Gated Recurrent Unit Network (2018, background): “In recent years, deep neural networks (DNNs) – a branch of machine learning has achieved numerous successes in various domains such as speech recognition [15], computer vision [16].”
A survey of deep neural network architectures and their applications (2017, background): “achievements have also been obtained in ASR, especially in distant talking situations [132,161,164], audio-visual speech recognition (AVSR) systems [125], and data augmentation on the basis of labelpreserving transformations [27].”
A Survey of Deep Learning and Its Applications: A New Paradigm to Machine Learning (2019, background): “Noda et al. [40] proposed a novel utilization of deep neural networks in audio visual speech recognition.”
Combining Residual Networks with LSTMs for Lipreading (2017, methodology): “More recent works deploy deep learning methods either for extracting ”deep” features ([19] [20] [21]) or for building end-to-end architectures.”
Lipreading Using Temporal Convolutional Networks (2020, methodology): “The same two-step approach was also followed in the first deep learning works, where the feature extraction step was replaced by deep autoencoders and HMMs were replaced by Long-Short Term Memory (LSTM) networks [4, 5, 6].”
Deep Audio-Visual Speech Recognition (2018, methodology): “The use of HMMs together with hand-crafted or pretrained visual features have proved popular – [48] encodes input images using DBF; [20] used DCT; and [38] uses a CNN pre-trained to classify phonemes; all three combine these features with HMMs to classify spoken digits or isolated words.”
Learning Weakly Supervised Multimodal Phoneme Embeddings (2017, methodology): “Classical audio-visual ASR systems use an audio-visual corpus to build complex supervised classifiers able to have an efficient phoneme representation [10, 11].”
A comprehensive review on ensemble deep learning: Opportunities and challenges (2023, background): “…architectures have shown great success in almost all challenges related to machine learning across different areas, such as NLP (Mohammed and Kora, 2019; Elnagar et al., 2020), computer vision (Haque et al., 2020; Brunetti et al., 2018), speech recognition (Jaouedi et al., 2020; Noda et al., 2015).”
Lip Reading Sentences in the Wild (2016, methodology): “The use of HMMs together with hand-crafted or pre-trained visual features have proved popular – [36] encodes input images using DBF; [14] used DCT; and [28] uses a CNN pre-trained to classify phonemes; all three combine these features with HMMs to classify spoken digits or isolated words.”