Analyzing Hidden Representations in End-to-End Automatic Speech Recognition Systems

By Yonatan Belinkov, James R. Glass. 2017

98 citations recorded by the source corpus.

Neural models have become ubiquitous in automatic speech recognition systems. While neural networks are typically used as acoustic models in more complex systems, recent studies have explored end-to-end speech recognition systems based on neural networks, which can be trained to directly predict text from input acoustic features. Although such systems are conceptually elegant and simpler than traditional systems, it is less obvious how to interpret the trained models. In this work, we analyze the speech representations learned by a deep end-to-end model that is based on convolutional and recurrent layers, and trained with a connectionist temporal classification (CTC) loss. We use a pre-trained model to generate frame-level features which are given to a classifier that is trained on frame classification into phones. We evaluate representations from different layers of the deep model and compare their quality for predicting phone labels. Our experiments shed light on important aspects of the end-to-end model such as layer depth, model complexity, and other design choices.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

Probing Classifiers: Promises, Shortcomings, and Advances (2021, background): “7 “evaluate the quality of the trained classiﬁer on the given task as a proxy to the quality of the extracted representations” (Belinkov et al. 2017a).”
Designing and Interpreting Probes with Control Tasks (2019, methodology): “Applying dropout, the most popular probing regularization method (Adi et al., 2017; Belinkov and Glass, 2019; Şahin et al., 2019; Kim et al., 2019; Elloumi et al., 2018; Belinkov and Glass, 2017; Belinkov et al., 2018) does not consistently lead to high-accuracy, high-selectivity MLP probes across a”
Information-Theoretic Probing for Linguistic Structure (2020, background): “These functions have the effect of decontextualizing the embeddings, being related to the common trend of analyzing probe results in comparison to input layer embeddings (Belinkov and Glass, 2017; Liu et al., 2019; Hewitt and Manning, 2019; Tenney et al., 2019).”
Analyzing analytical methods: The case of phonology in neural models of spoken language (2020, unknown): “Similarly, Belinkov and Glass (2017) use diagnostic classiﬁers to study the encoding of phonemes in an end-to-end ASR system with convolutional and recurrent layers, by feeding local (frame-based) representations to an MLP to predict a phoneme label.”
AGI Safety Literature Review (2018, background): “Looking beyond RL, Olah et al. (2017) and Jaakkola, 2017; Belinkov and Glass, 2017; Lei et al., 2016).”
A comprehensive analysis on attention models (2019, methodology): “2 Related work Belinkov and Glass [2017] provides an analysis of the hidden activations TODO ... Submitted to 32nd Conference on Neural Information Processing Systems (NIPS 2018).”
Probing Classifiers: Promises, Shortcomings, and Alternatives (2021, background): “An analysis of phonetic information in layers of a speech recognition systems (Belinkov and Glass 2017) partly motivated Krishna, Toshniwal, and Livescu (2018) to propose multi-task learning with phonetic supervision on intermediate layers.”
Analyzing Learned Representations of a Deep ASR Performance Prediction Model (2018, methodology): “Following a methodology similar to (Belinkov and Glass, 2017), our deep performance prediction model is used to generate utterance level features that are given to a shallow classifier”
On internal language representations in deep learning: an analysis of machine translation and speech recognition (2018, background): “The work presented in Chapter 5 was published in [29].”
Hierarchical Multitask Learning for CTC-based Speech Recognition (2018, background): “for both tasks. These results – in particular, the worse WER at i= 1 and the monotonic improvement in PER with i– may seem to be in contrast with earlier work analyzing learned neural representations [3], which reported that the ﬁrst layer of a deep CTC recognizer (specifically, DeepSpeech 2 [31]) i”
Analyzing ASR Pretraining for Low-Resource Speech-to-Text Translation (2019, methodology): “We follow [35] and [10], who built diagnostic classifiers [36] to examine the representation of phonetic information in end-to-end ASR and AST systems, respectively.”