Recent advances in the automatic recognition of audiovisual speech

By Gerasimos Potamianos, Chalapathy Neti, Guillaume Gravier, Ashutosh Garg, Andrew W. Senior. Proceedings of the IEEE, 2003

819 citations recorded by the source corpus.

Subfields: Signal Processing

View Paper: https://www.semanticscholar.org/paper/3326838b53788bdb79d3b37b3ddad6a619ce53b1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Signal Processing

Citations in the corpus

Rate-Invariant Analysis of Trajectories on Riemannian Manifolds with Application in Visual Speech Recognition (2014, methodology): “Hidden Markov model (HMM) has often been used to perform VSR, e.g. [12].”
Multimodal Machine Learning: A Survey and Taxonomy (2017, methodology): “ late and hybrid fusion. In contrast, late fusion uses unimodal decision values and fuses them using a fusion mechanism such as averaging [181], voting schemes [144], weighting based on channel noise [163] and signal variance [53], or a learned model [68], [168]. It allows for the use of different m”
Deep Audio-visual Learning: A Survey (2020, methodology): “on. Fig. 5. Earlier efforts on audio-visual fusion models usually consisted of two steps: 1) extracting features from the image and audio signals and 2) combining the features for joint classiﬁcation [62, 63, 64]. Later, taking advantage of deep learning, feature extraction was replaced with a neura”
Multi-modal Self-Supervision from Generalized Data Transformations (2020, neutral): “lti-modal learning has been used for several applications such as lip reading [4, 20, 21], audio-visual source separation and localization [3, 7, 28, 29, 39, 77, 87, 91, 113, 114], speech recognition [2, 73, 84], efﬁcient inference [30, 56], egocentric action recognition [51] and navigation [16]. Th”
Combining Residual Networks with LSTMs for Lipreading (2017, neutral): “Prior to the advent of deep learning ([10]) most of the work in lipreading was based on hand-engineered features, that were usually modeled by HMM-based pipeline, [11] [12] [13] [14] [15].”
Multimodal fusion for multimedia analysis: a survey (2010, background): “Existing surveys in this direction are mostly focused on a particular aspect of the analysis task, such as multimodal video indexing [26, 120]; automatic audio-visual speech recognition [106]; biometric audiovisual speech synchrony [20]; multi-sensor management for information fusion [146]; face rec”
Trusted Multi-View Classification With Dynamic Evidential Fusion (2022, methodology): “The naive fusion strategy is the most widely used method, which fuses decision results from different sources through decision averaging [12], [51], [52], decision voting [53], [54], or weighting multiple decisions [55], [56], [57], [58].”
Audio-Visual Speaker Diarization Based on Spatiotemporal Bayesian Fusion (2016, background): “, [3], [13], [19], [22], [23], [24] require long sequences of audiovisual data, hence they can only be used offline such as the analysis of broadcast news, of audiovisual conferences, etc.”
Handbook of Face Recognition, 2nd Edition (2011, background): “In Japan privacy concerns led to Street View imagery being recaptured with the car-mounted cameras lowered by 40cm so that the service would not present imagery taken over people’s garden walls, and there has been considerable opposition to the service on privacy grounds in Switzerland and Germany [”
End-To-End Audio-Visual Speech Recognition with Conformers (2021, methodology): “Traditional audio-visual speech recognition methods follow a two-step approach, feature extraction and recognition [9, 26].”
LipNet: End-to-End Sentence-level Lipreading (2016, methodology): “Machine lipreading is difﬁcult because it requires extracting spatiotemporal features from the video (since both position and motion are important).”
Adaptive Multimodal Fusion by Uncertainty Compensation With Application to Audiovisual Speech Recognition (2009, methodology): “…the integral in (2) is in general intractable, we can obtain a closed-form solution in the important special case of Gaussian data model, , with Gaussian observation noise, , where stands for the multivariate Gaussian probability density function on with mean and covariance matrix .”
Audiovisual SlowFast Networks for Video Recognition (2020, background): “Other audiovisual tasks that have been studied include audio-visual speech recognition [54, 50], sound-source localization [3, 59], audio-visual source separation [52, 19], and audiovisual question answering [1].”
Weakly Supervised Learning with Multi-Stream CNN-LSTM-HMMs to Discover Sequential Parallelism in Sign Language Videos (2020, background): “A good overview can be found in [17].”
Self-Supervised Speech Representation Learning: A Review (2022, background): “Multi-modal work includes supervised settings, such as audio-visual ASR [143], [144] and person identification [145] which have been studied for decades.”