Twin Regularization for online speech recognition

By Mirco Ravanelli, Dmitriy Serdyuk, Yoshua Bengio. 2018

16 citations recorded by the source corpus.

Online speech recognition is crucial for developing natural human-machine interfaces. This modality, however, is significantly more challenging than off-line ASR, since real-time/low-latency constraints inevitably hinder the use of future information, that is known to be very helpful to perform robust predictions. A popular solution to mitigate this issue consists of feeding neural acoustic models with context windows that gather some future frames. This introduces a latency which depends on the number of employed look-ahead features. This paper explores a different approach, based on estimating the future rather than waiting for it. Our technique encourages the hidden representations of a unidirectional recurrent network to embed some useful information about the future. Inspired by a recently proposed technique called Twin Networks, we add a regularization term that forces forward hidden states to be as close as possible to cotemporal backward ones, computed by a "twin" neural network running backwards in time. The experiments, conducted on a number of datasets, recurrent architectures, input features, and acoustic conditions, have shown the effectiveness of this approach. One important advantage is that our method does not introduce any additional computation at test time if compared to standard unidirectional recurrent networks.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

Interpretable Convolutional Filters with SincNet (2018, methodology): “Multiple standard convolutional, fully-connected or recurrent layers [15, 16, 17, 45] can then be stacked together to finally perform a classification with a softmax classifier.”
Knowledge Distillation from Offline to Streaming RNN Transducer for End-to-End Speech Recognition (2020, methodology): “For attention-based encoder-decoder and hybrid RNN/HMM, twin regularization to make forward hidden states as close as backward hidden states has been applied [30,31].”
Future-Guided Incremental Transformer for Simultaneous Translation (2020, unknown): “To utilize future source information to enhance the predictive ability, we encourage the model to embed some future information through knowledge distillation (Hinton, Vinyals, and Dean 2015; Ravanelli, Serdyuk, and Bengio 2018; Novitasari et al. 2019).”
Speaker Recognition from Raw Waveform with SincNet (2018, methodology): “Multiple standard convolutional, fully-connected or recurrent layers [37–40] can then be stacked together to finally perform a speaker classification with a softmax classifier.”
The Pytorch-kaldi Speech Recognition Toolkit (2018, background): “PyTorch-Kaldi natively supports several DNNs, CNNs, and RNNs models.”