Transformers with convolutional context for ASR

By Abdel-rahman Mohamed, Dmytro Okhonko, Luke S. Zettlemoyer. 2019

174 citations recorded by the source corpus.

The recent success of transformer networks for neural machine translation and other NLP tasks has led to a surge in research work trying to apply it for speech recognition. Recent efforts studied key research questions around ways of combining positional embedding with speech features, and stability of optimization for large scale learning of transformer networks. In this paper, we propose replacing the sinusoidal positional embedding for transformers with convolutionally learned input representations. These contextual representations provide subsequent transformer blocks with relative positional information needed for discovering long-range relationships between local concepts. The proposed system has favorable optimization characteristics where our reported results are produced with fixed learning rate of 1.0 and no warmup steps. The proposed model achieves a competitive 4.7% and 12.9% WER on the Librispeech ``test clean'' and ``test other'' subsets when no extra LM text is provided.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Imputer: Sequence Modelling via Imputation and Dynamic Programming (2020, methodology): “Mohamed et al. (2019) also combined convolutions with self-attention, where they also use convolutions to model local acoustic features, our work also uses self-attention to capture global signals from both the acoustics and the previous alignment, which licenses powerful bidirectional language…”
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (2020, methodology): “Instead of fixed positional embeddings which encode absolute positional information, we use a convolutional layer similar to [37, 4, 57] which acts as relative positional embedding.”
Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq (2020, background): “Table 2 shows the test BLEU of our Transformer-based models (“T-Sm” and “Multi.”
Iterative Pseudo-Labeling for Speech Recognition (2020, positive): “Some of the most promising are from the Transformer family [1, 2, 3, 4, 5], which give state-of-the-art results on many ASR benchmarks and close the gap between end-to-end and hybrid systems.”
Linformer: Self-Attention with Linear Complexity (2020, background): “Transformer models (Vaswani et al., 2017) have become ubiquitous for wide variety of problems in natural language processing (NLP), including translation (Ott et al., 2018), text classiﬁcation, question answering, among others (Raffel et al., 2019; Mohamed et al., 2019).”
Generative Pre-Training for Speech with Autoregressive Predictive Coding (2019, methodology): “Regarding the backbone architecture of APC, the Transformer model can be potentially improved by modifying the way we inject positional information [31, 32].”
Transformer-Based Acoustic Modeling for Hybrid Speech Recognition (2019, methodology): “Following [27], we use convolution layers as pre-processors for the transformer layer’s input and compare it with other positional encoding methods in Section 4.2.”
DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization (2020, background): “This performs an implicit relative positional encoding as pointed out in [32].”
Random Feature Attention (2021, background): “Answering to this, many approaches have been proposed, e.g., learning the attention spans (Sukhbaatar et al., 2019; Wu et al., 2020), and enhancing the attention computation with recurrent (Hao et al., 2019; Chen et al., 2019) or convolutional (Wu et al., 2019; Mohamed et al., 2019) components.”
Self-Training for End-to-End Speech Translation (2020, methodology): “The Transformer architecture, VGGTRANSFORMER, is an adaptation of Transformer to the ASR task [21].”
A Convolutional Deep Markov Model for Unsupervised Speech Representation Learning (2020, methodology): “This convolutional architecture is inspired by [22], but other acoustic models such as TimeDepth Separable Convolutions [24], VGG transformer [25], or ResDAVENet [12] could be used here.”
RobuTrans: A Robust Transformer-Based Text-to-Speech Model (2020, methodology): “Instead of adding this positional embedding, as investigated in the speech recognition community (Mohamed, Okhonko, and Zettlemoyer 2019), we can simply remove the position embedding and count on the CNN used in Encoder Pre-net to model the relative position information in a ﬁxed window.”
BENDR: Using Transformers and a Contrastive Self-Supervised Learning Task to Learn From Massive Amounts of EEG Data (2021, methodology): “While it seems that convolutional position encodings offer better performance [62], this input width exceeded the entire length of all but the sleep classiﬁcation task (the length we chose was optimized for pre-training behaviour).”