Self-Attentional Acoustic Models

By Matthias Sperber, Jan Niehues, Graham Neubig, Sebastian Stüker, A. Waibel. 2018

160 citations recorded by the source corpus.

Self-attention is a method of encoding sequences of vectors by relating these vectors to each-other based on pairwise similarities. These models have recently shown promising results for modeling discrete sequences, but they are non-trivial to apply to acoustic modeling due to computational and modeling issues. In this paper, we apply self-attention to acoustic modeling, proposing several improvements to mitigate these issues: First, self-attention memory grows quadratically in the sequence length, which we address through a downsampling technique. Second, we find that previous approaches to incorporate position information into the model are unsuitable and explore other representations and hybrid models to this end. Third, to stress the importance of local context in the acoustic signal, we propose a Gaussian biasing approach that allows explicit control over the context range. Experiments find that our model approaches a strong baseline based on LSTMs with network-in-network connections while being much faster to compute. Besides speed, we find that interpretability is a strength of self-attentional acoustic models, and demonstrate that self-attention heads learn a linguistically plausible division of labor.

Subfields: Artificial Intelligence, Model Architectures, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Speech & Audio

In Signal Processing

Citations in the corpus

Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss (2020, methodology): “Transformer-based alternatives to RNNs have recently been explored for use in ASR [9, 10, 11, 12].”
Transformers with convolutional context for ASR (2019, background): “There has been recent research interest in using transformer networks for end-to-end ASR both with CTC loss [13] and in an encoder-decoder framework [14, 15] with modest performance compared to baseline systems.”
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention (2020, background): “, 2019), audio (Sperber et al., 2018), and images (Parmar et al.”
Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders (2019, methodology): “As direct addition of acoustic features to positional encoding may lead to potential training failure [14], the input frames are ﬁrst projected linearly to the hidden dimension of H dim before adding with positional encoding [15].”
Transformer-Based Acoustic Modeling for Hybrid Speech Recognition (2019, background): “In [18], various positional embedding meth-ods were investigated for a sequence-to-sequence model, where it is found that replacing the FFN network with a LSTM layer to make the self-attention layer position aware yielded better performance.”
On the limit of English conversational speech recognition (2021, unknown): “Self-attention based transformer models have been explored as encoder and also as decoder for ASR in [29–32].”
A Comparison of Transformer and LSTM Encoder Decoder Models for ASR (2019, methodology): “More recent work suggested to use the Transformer directly as the acoustic model [34] or as an end-to-end ASR model [34–42].”
Convolutional Self-Attention Networks (2019, methodology): “Comparing with previous work on modeling locality for SANs (e.g. Shaw et al., 2018; Yang et al., 2018; Sperber et al., 2018), our model boosts performance on both translation quality and training efﬁciency.”
Dual-Path Transformer Network: Direct Context-Aware Modeling for End-to-End Monaural Speech Separation (2020, methodology): “To learn the order information, we replace the first fully connected layer with a recurrent neural network in the feed-forward network, which is a interesting improvement from paper [22]:”
Attention in Natural Language Processing (2019, background): “other different fields, such as speech recognition [19]–[21],”
Self-attention Networks for Connectionist Temporal Classification in Speech Recognition (2019, methodology): “In the first layers, we similarly observe a differentiation of variances, along with wide-context heads; in later layers, unlike [27] we still see mild differentiation of variances.”
Convolutional Self-Attention Network (2018, background): “• From the attention distribution perspective, “NEIGHBOR” (Sperber et al., 2018) and “LOCAL HARD” (Luong et al.”
Enhancing Transformer for End-to-end Speech-to-Text Translation (2019, methodology): “Given the results of Transformer in MT, recent works on ASR proposed SANs for both acoustic modeling (Sperber et al., 2018; Povey et al., 2018) and end-to-end ASR (Dong et al., 2018; Zhou et al., 2018a; Zhou et al., 2018b).”
Multilingual Sequence-to-Sequence Speech Recognition: Architecture, Transfer Learning, and Language Modeling (2018, methodology): “Considering the limitations of attention in performing monotonic alignment [19, 20], we choose to use CTC loss function to aid the attention mechanism in both training and decoding.”
Earthquake transformer—an attentive deep-learning model for simultaneous earthquake detection and phase picking (2020, methodology): “One LSTM layer with 16 units is applied before the ﬁ rst attention block at each level to provide position information 42,51 .”
Recent Advances in End-to-End Automatic Speech Recognition (2021, methodology): “Another popular method is Gaussian soft-mask [262] which changes the softmax calculation in self-attention in Equation (12) as”
Towards Online End-to-end Transformer Automatic Speech Recognition (2019, methodology): “A simple way to process the encoder online is blockwise computation, as in [17, 19, 23].”