Lexicon-Free Conversational Speech Recognition with Neural Networks

By Andrew L. Maas, Ziang Xie, Dan Jurafsky, Andrew Y. Ng. 2015

146 citations recorded by the source corpus.

We present an approach to speech recognition that uses only a neural network to map acoustic input to characters, a character-level language model, and a beam search decoding procedure. This approach eliminates much of the complex infrastructure of modern speech recognition systems, making it possible to directly train a speech recognizer using errors generated by spoken language understanding tasks. The system naturally handles out of vocabulary words and spoken word fragments. We demonstrate our approach using the challenging Switchboard telephone conversation transcription task, achieving a word error rate competitive with existing baseline systems. To our knowledge, this is the first entirely neural-network-based system to achieve strong speech transcription results on a conversational speech task. We analyze qualitative differences between transcriptions produced by our lexicon-free approach and transcriptions produced by a standard speech recognition system. Finally, we evaluate the impact of large context neural network character language models as compared to standard n-gram models within our framework.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

Deep Speech 2 : End-to-End Speech Recognition in English and Mandarin (2015, methodology): “, 2006) coupled with an RNN to model temporal information also performs well in endto-end speech recognition with character outputs (Graves & Jaitly, 2014a; Hannun et al., 2014b;a; Maas et al., 2015).”
Listen, attend and spell: A neural network for large vocabulary conversational speech recognition (2015, positive): “There has recently been an explosion in methods for end-to-end trained speech models because of their inherent simplicity compared to current speech recognition systems [2, 24, 6, 25, 14].”
ESPnet: End-to-End Speech Processing Toolkit (2018, methodology): “• CTC-based: EESEN [11], Stanford CTC [28], Baidu Deepsppech [12],”
EESEN: End-to-end speech recognition using deep RNN models and WFST-based decoding (2015, methodology): “THE EESEN FRAMEWORK: DECODING”
Improved training of end-to-end attention models for speech recognition (2018, neutral): “tional data. model LM label WER[%] unit SWB CH Hub5’01 LF MMI, 2016 [7] 4-gram CDp 9.6 19.3 hybrid 4-gram CDp 9.8 19.0 14.7 hybrid LSTM CDp 8.3 17.3 12.9 CTC1, 2014 [12] RNN chars 20.0 31.8 CTC, 2015 [60] none chars 38.0 56.1 CTC, 2015 [60] RNN chars 21.4 40.2 attention, 2016 [61] none chars 32.8 52”
Who Needs Words? Lexicon-Free Speech Recognition (2019, methodology): “To the best of our knowledge, the first ASR system to achieve competitive results with a character-based LM and without a lexicon (on Switchboard and WSJ) was [20], that our lexicon-free character-based ConvLM surpasses on WSJ (see Table 4).”
Speech Recognition via Large-Scale Weak Supervision (unknown): “…and by 2012 the RNN-Transducer was deﬁned and applied to phone recognition (Graves 2012, Graves et al. 2013), and then to end-to-end speech recognition rescoring (Graves and Jaitly, 2014), and then recognition (Maas et al., 2015), with advances such as specialized beam search (Hannun et al., 2014).”
Segmental Recurrent Neural Networks (2015, background): “CTC has demonstrated impressive success in various fully discriminative end-to-end speech recognition models (Graves & Jaitly, 2014; Maas et al., 2015; Hannun et al., 2014, inter alia ).”
Direct Acoustics-to-Word Models for English Conversational Speech Recognition (2017, background): “In particular, the CTC loss enables training AMs by mapping acoustics to phone or character sequences without any frame-level alignment [11–17].”
Multi-Modal Data Augmentation for End-to-end ASR (2018, background): “[7, 8] incorporate a character level language model during beam search, possibly disallowing character sequences absent from a dictionary, while [9] includes a full word level language model in decoding by simultaneously keeping track of word histories and word prefixes.”
Maximum a posteriori Based Decoding for CTC Acoustic Models (2016, methodology): “Recently, the connectionist temporal classification (CTC) [16], which enables a neural network to learn the direct mapping from audio input X to symbol sequence s, has been successfully applied to large vocabulary continuous speech recognition (LVCSR) tasks [17, 18, 19, 20, 21, 22, 23].”
Scaling data-driven robotics with reward sketching and batch reinforcement learning (2019, background): “Deep learning has successfully advanced many areas of artificial intelligence, including vision [39, 26], speech recognition [24, 46, 4], natural language processing [17], and reinforcement learning (RL) [49, 63].”
Word-level Speech Recognition with a Dynamic Lexicon (2019, methodology): “Unlike lexicon-free approaches [22, 25, 33] our model requires a lexicon.”
Character-based Neural Machine Translation (2015, background): “For example, speech recognition models (Chan et al., 2015; Maas et al., 2015) and language modeling (Sutskever et al., 2011; Mikolov et al., 2012).”
Very Deep Self-Attention Networks for End-to-End Speech Recognition (2019, methodology): “d and endto-end systems reporting on Hub5’00 test set with 300h SWB training set. Hybrid/End-to-End Models Tgt Unit SWB CH TDNN +LFMMI [23] Phone 10.0 20.1 BLSTM +LFMMI [23] Phone 9.6 19.3 CTC+CharLM [24] Char 21.4 40.2 LSTM w/attention [1] Char 15.8 36.0 Iterated-CTC +LSTM-LM [25] Char 14.0 25.3 Se”
On internal language representations in deep learning: an analysis of machine translation and speech recognition (2018, methodology): “Such models have recently become popular in the ASR community thanks to their simple and elegant architecture [18, 59, 70, 130, 222, 234].”