Hierarchical Multitask Learning for CTC-based Speech Recognition

By Kalpesh Krishna, Shubham Toshniwal, Karen Livescu. 2018

45 citations recorded by the source corpus.

Previous work has shown that neural encoder-decoder speech recognition can be improved with hierarchical multitask learning, where auxiliary tasks are added at intermediate layers of a deep encoder. We explore the effect of hierarchical multitask learning in the context of connectionist temporal classification (CTC)-based speech recognition, and investigate several aspects of this approach. Consistent with previous work, we observe performance improvements on telephone conversational speech recognition (specifically the Eval2000 test sets) when training a subword-level CTC model with an auxiliary phone loss at an intermediate layer. We analyze the effects of a number of experimental variables (like interpolation constant and position of the auxiliary loss function), performance in lower-resource settings, and the relationship between pretraining and multitask learning. We observe that the hierarchical multitask approach improves over standard multitask training in our higher-data experiments, while in the low-resource settings standard multitask training works well. The best results are obtained by combining hierarchical multitask learning and pretraining, which improves word error rates by 3.4% absolute on the Eval2000 test sets.

Subfields: Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Speech & Audio

Citations in the corpus

Probing Classifiers: Promises, Shortcomings, and Advances (2021, background): “An analysis of phonetic information in layers of a speech recognition systems (Belinkov and Glass 2017) partly motivated Krishna, Toshniwal, and Livescu (2019) to propose multi-task learning with phonetic supervision on intermediate layers.”
Probing Classifiers: Promises, Shortcomings, and Alternatives (2021, background): “An analysis of phonetic information in layers of a speech recognition systems (Belinkov and Glass 2017) partly motivated Krishna, Toshniwal, and Livescu (2018) to propose multi-task learning with phonetic supervision on intermediate layers.”
Massively Multilingual Adversarial Speech Recognition (2019, methodology): “We performed limited preliminary experiments to determine how best to use the phoneme objec-tive, which corroborated recent work in hierarchical training objectives that supports inserting the phoneme objective in the layers below the ﬁnal layer (Krishna et al., 2018).”
A comprehensive analysis on attention models (2019, methodology): “…et al., 2014, Luong et al., 2015, Wu et al., 2016] has been successfully applied to speech recognition [Chan et al., 2015, Chiu et al., 2017, Toshniwal et al., 2018, Krishna et al., 2018, Zeyer et al., 2018b] and is one promising end-to-end approach, i.e. the models output are words,…”