Multi-Modal Data Augmentation for End-to-end ASR
By Adithya Renduchintala, Shuoyang Ding, Matthew Wiesner, Shinji Watanabe. 2018
40 citations recorded by the source corpus.
We present a new end-to-end architecture for automatic speech recognition (ASR) that can be trained using \emph{symbolic} input in addition to the traditional acoustic input. This architecture utilizes two separate encoders: one for acoustic input and another for symbolic input, both sharing the attention and decoder parameters. We call this architecture a multi-modal data augmentation network (MMDA), as it can support multi-modal (acoustic and symbolic) input. The MMDA architecture attempts to eliminate the need for an external LM, by enabling seamless mixing of large text datasets with significantly smaller transcribed speech corpora during training. We study different ways of transforming large text corpora into a symbolic form suitable for training our MMDA network. Our best MMDA setup obtains small improvements on CER and achieves 8-10\% relative WER improvement on the WSJ data set.
Subfields: Artificial Intelligence, Speech & Audio
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Speech & Audio
Citations in the corpus
- A Survey of Data Augmentation Approaches for NLP (2021, methodology): “Wiesner et al. (2018) and Renduchintala et al. (2018) propose DA methods for end-to-end automatic speech recognition (ASR).”
- Self-supervised Sequence-to-sequence ASR using Unpaired Speech and Text (2019, methodology): “In the latter model, an encoder component is shared to have common representation across both speech and text data [11].”
- Self-Supervised Speech Representation Learning: A Review (2022, background): “, encoder features [319] and phoneme features [320], [321].”
- Semi-Supervised Sequence-to-Sequence ASR Using Unpaired Speech and Text (2019, methodology): “In the latter model, an encoder component is shared to have common representation across both speech and text data [11].”
- Low Resource Multi-modal Data Augmentation for End-to-end ASR (2018, methodology): “[14] bypasses this problem by apply to each phoneme in a non-English language the duration model corresponding to a close English phone equivalent.”
- A Comparison of Techniques for Language Model Integration in Encoder-Decoder Speech Recognition (2018, methodology): “A workaround of “translating” the text to phoneme sequences and using the resulting paired data in a multitask learning setup has been explored by Ren-duchintala et al. [14].”
- Back-Translation-Style Data Augmentation for end-to-end ASR (2018, methodology): “Instead of using a text-to-speech system on unpaired text to produce synthetic speech [18] or using grapheme to phoneme conversion to generate paired text and pseudo speech sequences based on phonemes [19], we build a text-to-encoder model which learns to predict the hidden states of the E2E-ASR enc”
- Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task (2021, background): “The input text tokens are converted into their corresponding pronunciation form as phoneme sequences (Tang et al., 2021; Renduchintala et al., 2018).”
- End-to-End Speech Recognition: A Survey (2023, background): “Another prominent feature of E2E systems besides the alignment property is their direct character-level modeling avoiding phoneme-based modeling and pronunciation lexica [16], [19], [74], [75], [76], [77], [78], [79], [80], [81], [82], with some even heading for whole-word modeling [30], [76].”
- Leveraging Weakly Supervised Data to Improve End-to-end Speech-to-text Translation (2018, methodology): “Instead of using TTS, [18] synthesizes repeated phoneme sequence from unlabeled text to mimic temporal ar X iv :1 81 1.”