Extracting deep bottleneck features using stacked auto-encoders

By Jonas Gehring, Yajie Miao, Florian Metze, A. Waibel. IEEE International Conference on Acoustics, Speech, and Signal Processing, 2013

307 citations recorded by the source corpus.

In this work, a novel training scheme for generating bottleneck features from deep neural networks is proposed. A stack of denoising auto-encoders is first trained in a layer-wise, unsupervised manner. Afterwards, the bottleneck layer and an additional layer are added and the whole network is fine-tuned to predict target phoneme states. We perform experiments on a Cantonese conversational telephone speech corpus and find that increasing the number of auto-encoders in the network produces more useful features, but requires pre-training, especially when little training data is available. Using more unlabeled data for pre-training only yields additional gains. Evaluations on larger datasets and on different system setups demonstrate the general applicability of our approach. In terms of word error rate, relative improvements of 9.2% (Cantonese, ML training), 9.3% (Tagalog, BMMI-SAT training), 12% (Tagalog, confusion network combinations with MFCCs), and 8.7% (Switchboard) are achieved.

Subfields: Artificial Intelligence, Signal Processing

View Paper: https://www.semanticscholar.org/paper/2c5ee8c30bba238fbcb31456b10ebb2cdb8d1a35

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Mobile sensor data anonymization (2018, background): “The information bottleneck [4, 10] in the middle layer of the CAE helps the encoding part of the CAE to capture the main factors of variation in the data into this latent representation.”
Improving low-resource CD-DNN-HMM using dropout and multilingual DNN training (2013, background): “There has been a number of works dedicated to training multilingual bottleneck features [11, 12, 13, 14] or probabilistic posterior features [5, 12] used in tandem systems.”
Speaker Adaptive Training of Deep Neural Network Acoustic Models Using I-Vectors (2015, methodology): “Following our Deep BNF (DBNF) architecture [55], [56], the BNF-DNN has 6 hidden layers in which the 5th layer is a bottleneck layer.”
Deep neural networks employing Multi-Task Learning and stacked bottleneck features for speech synthesis (2015, background): “Bottleneck features have been extensively used in speech recognition [18, 19, 20].”
Gate Activation Signal Analysis for Gated Recurrent Neural Networks and its Correlation with Phoneme Boundaries (2017, methodology): “Autoencoder structures have been used for extracting bottleneck features [12], while GRNN with various structures can be learned very well without labeled data.”
Survey on Deep Neural Networks in Speech and Vision Systems (2019, background): “SAEs likewise are shown to outperform (GMM)-HMM on Cantonese and other speech recognition tasks [43].”
A deep auto-encoder based low-dimensional feature extraction from FFT spectral envelopes for statistical parametric speech synthesis (2016, methodology): “Deep auto-encoder based bottleneck features have been used by several groups for ASR [15, 16] and a deep denoising autoencoder has also verified for noise-robust ASR [17] or reverberant ASR tasks [18, 19].”
A Deep Learning Approach to Data-driven Parameterizations for Statistical Parametric Speech Synthesis (2014, methodology): “An example of this is the Deep Bottleneck Features that are used in Speech Recognition[17][18].”
A Review of Deep Learning Based Speech Synthesis (2019, methodology): “In these applications, RBM is often used for pre-training of deep auto-encoders (DAEs) [41,42] or DNNs.”
Anomaly Detection with Robust Deep Autoencoders (2017, background): “Denoising autoencoders [9, 17, 23, 30, 31], require access to a source of clean, noise-free data for training, and such data is not always readily available in real-world problems [28].”
Deep maxout networks for low-resource speech recognition (2013, methodology): “In tandem systems, we use DNNs to generate phone posteriors or bottleneck features (BNF), and build normal GMMHMM models with the discriminative front-end [3, 4, 5, 6].”