Small-Footprint Deep Neural Networks with Highway Connections for Speech Recognition

By Liang Lu, S. Renals. 2015

23 citations recorded by the source corpus.

For speech recognition, deep neural networks (DNNs) have significantly improved the recognition accuracy in most of benchmark datasets and application domains. However, compared to the conventional Gaussian mixture models, DNN-based acoustic models usually have much larger number of model parameters, making it challenging for their applications in resource constrained platforms, e.g., mobile devices. In this paper, we study the application of the recently proposed highway network to train small-footprint DNNs, which are {\it thinner} and {\it deeper}, and have significantly smaller number of model parameters compared to conventional DNNs. We investigated this approach on the AMI meeting speech transcription corpus which has around 70 hours of audio data. The highway neural networks constantly outperformed their plain DNN counterparts, and the number of model parameters can be reduced significantly without sacrificing the recognition accuracy.

Subfields: Artificial Intelligence, Efficiency & Compression, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Efficiency & Compression

In Speech & Audio

In Signal Processing

Citations in the corpus

Light Gated Recurrent Units for Speech Recognition (2018, neutral): “ is also around 30%. The reduction of the computational complexity, originated by a more compact model, also arises for testing purposes, making our model potentially suitable for smallfootprint ASR, [70]–[75], which studies DNNs designed for portable devices with small computational capabilities. V”
Composition of Deep and Spiking Neural Networks for Very Low Bit Rate Speech Coding (2016, background): “ple [74]–[77]) showed a possible 80% memory reduction.”
Knowledge Distillation for Sequence Model (2018, background): “Thus researches have been conducted on transferring the learned models to lowresource platforms [3, 4, 5, 6].”