Segmental Recurrent Neural Networks for End-to-End Speech Recognition

By Liang Lu, Lingpeng Kong, Chris Dyer, Noah A. Smith, S. Renals. 2016

83 citations recorded by the source corpus.

We study the segmental recurrent neural network for end-to-end acoustic modelling. This model connects the segmental conditional random field (CRF) with a recurrent neural network (RNN) used for feature extraction. Compared to most previous CRF-based acoustic models, it does not rely on an external system to provide features or segmentation boundaries. Instead, this model marginalises out all the possible segmentations, and features are extracted from the RNN trained together with the segmental CRF. In essence, this model is self-contained and can be trained end-to-end. In this paper, we discuss practical training and decoding issues as well as the method to speed up the training in the context of speech recognition. We performed experiments on the TIMIT dataset. We achieved 17.3 phone error rate (PER) from the first-pass decoding --- the best reported result using CRFs, despite the fact that we only used a zeroth-order CRF and without using any language model.

Subfields: Artificial Intelligence, Model Architectures, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Speech & Audio

In Signal Processing

Citations in the corpus

Learning Filterbanks from Raw Speech for Phone Recognition (2017, background): “Dev PER Test PER Hybrid HMM/Hierarchical CNN + Maxout + Dropout [10] MFSC + energy + ∆ + ∆∆ 13.3 16.5 CNN + CRF on raw speech [13] wav - 29.2 CNN-Conv2D-10L-Maxout [ [10] are trained in an end-to-end fashion.”
Exploring neural transducers for end-to-end speech recognition (2017, methodology): “Segmental RNNs [18] provide another alternative way to model the ASR task.”
Quaternion Convolutional Neural Networks for End-to-End Automatic Speech Recognition (2018, result): “Other recent results with real-valued vectors of similar features are reported in [29, 4, 30, 31].”
Low Latency Acoustic Modeling Using Temporal Convolution and LSTMs (2018, methodology): “This architecture is similar to the hierarchical subsampling networks, proposed in [20] and more recently applied in [21] and [22].”
The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches (2018, methodology): “s Pooling[18] CTC[19] DCNN [20] Ensemble DNN/CNN/RNN[21] RNN transducer[19] Attention-based RNN[22] Segmental RNN[23] Phone error rate (PER) in percentage(%)”
Learning Neural Templates for Text Generation (2018, background): “Finally, there has also been much recent interest in segmental RNN models for non-generative tasks in NLP (Tang et al., 2016; Kong et al., 2016; Lu et al., 2016).”
Improving Human Text Comprehension through Semi-Markov CRF-based Neural Section Title Generation (2019, methodology): “While they were originally developed for information extraction (Sarawagi and Co-hen, 2005), it is most commonly applied to speech recognition within the acoustic model to improve segmentation between different words (He and Fosler-Lussier, 2015; Lu et al., 2016; Kong et al., 2015).”
Recurrent Neural Aligner: An Encoder-Decoder Neural Network Model for Sequence to Sequence Mapping (2017, methodology): “A segmental RNN model has been also proposed for speech recognition [12].”
Morphological Inflection Generation with Hard Monotonic Attention (2016, methodology): “Lu et al. (2016) used a combination of an RNN encoder with a CRF layer to model the dependencies in the output sequence.”