Multitask Learning with Low-Level Auxiliary Tasks for Encoder-Decoder Based Speech Recognition

By Shubham Toshniwal, Hao Tang, Liang Lu, Karen Livescu. 2017

124 citations recorded by the source corpus.

End-to-end training of deep learning-based models allows for implicit learning of intermediate representations based on the final task loss. However, the end-to-end approach ignores the useful domain knowledge encoded in explicit intermediate-level supervision. We hypothesize that using intermediate representations as auxiliary supervision at lower levels of deep networks may be a good way of combining the advantages of end-to-end training and more traditional pipeline approaches. We present experiments on conversational speech recognition where we use lower-level tasks, such as phoneme recognition, in a multitask training approach with an encoder-decoder model for direct character transcription. We compare multiple types of lower-level tasks and analyze the effects of the auxiliary tasks. Our results on the Switchboard corpus show that this approach improves recognition accuracy over a standard encoder-decoder model on the Eval2000 test set.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition (2019, neutral): “, (2017) [47] 23.”
Improved training of end-to-end attention models for speech recognition (2018, positive): “Recent investigations have shown promising results by applying the same approach for speech recognition [24–28].”
Analyzing Hidden Representations in End-to-End Automatic Speech Recognition Systems (2017, background): “For example, understanding representation learning at different layers of the end-to-end model can guide joint learning of phoneme recognition and ASR, as recently proposed in a multi-task learning framework [5].”
Beyond Shared Hierarchies: Deep Multitask Learning through Soft Layer Ordering (2017, methodology): “Several approaches integrate supervised feedback from each task at levels consistent with such a hierarchy (Hashimoto et al., 2016; Toshniwal et al., 2017; Zhang and Weiss, 2016) (Figure 1c).”
Mix-review: Alleviate Forgetting in the Pretrain-Finetune Framework for Neural Language Generation Models (2019, unknown): “Mix-review can also be viewed as a form of multi-task learning (Li et al., 2019), which has been shown to be useful in neural machine translation (NMT) (Niehues & Cho, 2017), speech recognition (Toshniwal et al., 2017), optical character recognition (OCR) (Liao et al., 2019), etc.”
Tied Multitask Learning for Neural Speech Translation (2018, methodology): “Toshniwal et al. (2017) apply multitask learning to neural speech recognition in a less traditional fashion: the lower-level outputs of the speech encoder are used for ﬁne-grained auxiliary tasks such as predicting HMM states or phonemes, while the ﬁnal output of the encoder is passed to a…”
Symbolic Inductive Bias for Visually Grounded Learning of Spoken Language (2018, methodology): “Toshniwal et al. (2017) use MTL for conversational speech recognition with lower-level tasks (e.g. phoneme recognition) in an encoder-decoder model for direct character transcription.”
Recent Advances in End-to-End Automatic Speech Recognition (2021, background): “It becomes the standard training recipe for most AED models [52, 95, 121, 127, 166, 277, 284, 300].”
Hierarchical Multitask Learning for CTC-based Speech Recognition (2018, methodology): “…loss is a convex combination of the two loss functions, using a tunable interpolation parameter λ : While we use a phone-level auxiliary task here, the hierarchical multitask learning idea is general and can be used with any relevant intermediate loss, or multiple intermediate losses (as in [8]).”
On internal language representations in deep learning: an analysis of machine translation and speech recognition (2018, background): “Understanding representation learning at different layers of the end-to-end model can guide joint learning of phone recognition and ASR, as recently proposed in a multi-task learning framework [320].”
AuxNet: Auxiliary tasks enhanced Semantic Segmentation for Automated Driving (2019, background): “In (Toshniwal et al., 2017), end2end speech recognition training uses auxiliary task phoneme recognition for initial stages.”