Speech-Transformer: A No-Recurrence Sequence-to-Sequence Model for Speech Recognition

By Linhao Dong, Shuang Xu, Bo Xu. 2018

1,188 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Natural Language Processing, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Signal Processing

Citations in the corpus

A Review of Sparse Expert Models in Deep Learning (2022, background): “Some examples are image classiﬁcation (Dosovitskiy et al., 2020), object detection (Carion et al., 2020), recommendation systems (Chen et al., 2019), speech recognition (Dong et al., 2018; Nakatani, 2019; Gulati et al., 2020).”
Imputer: Sequence Modelling via Imputation and Dynamic Programming (2020, methodology): “Our neural network architecture was also inspired by past work in convolutions (Sainath et al., 2013) and Transform-ers (Dong et al., 2018) for speech recognition.”
Unsupervised Speech Recognition (2021, background): “, 2015) has seen rapid improvement over the last few years due to advances in model architectures (Dong et al., 2018; Synnaeve et al., 2020; Gulati et al., 2020), semisupervised learning (Xu et al.”
Conformer: Convolution-augmented Transformer for Speech Recognition (2020, methodology): “This structure is also adopted by Transformer ASR models [7, 24].”
Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss (2020, methodology): “As typically used in sequence-to-sequence transduction tasks [4, 5, 6, 7, 8], Transformerbased models attend over encoder features using decoder features, implying that the decoding has to be done in a label-synchronous way, thereby posing a challenge for streaming speech recognition applications.”
Improving Transformer-based Speech Recognition Using Unsupervised Pre-training (2019, background): “Originally proposed as a better replacement of Recurrent Neural Network (RNN) for neural machine translation, Transformer based models got a lot of traction in the field of speech recognition recently [11, 12, 13, 14].”
A Comparison of Modeling Units in Sequence-to-Sequence Speech Recognition with the Transformer on Mandarin Chinese (2018, methodology): “The Transformer is chosen to be the basic architecture of sequence-to-sequence attention-based model in this paper [9, 10].”
Transformers without Tears: Improving the Normalization of Self-Attention (2019, methodology): “We compare the original learning rate schedule in equation 7 (I NV S QRT D ECAY ) with validation-based decay (V AL D ECAY ), possibly with no warmup (N O W ARMUP ).”
A Comparative Study on Transformer vs RNN in Speech Applications (2019, methodology): “Currently, existing Transformer-based speech applications [2]– [4] still lack an open source toolkit and reproducible experiments while previous studies in NMT [5], [6] provide them.”
Transformer-Based Acoustic Modeling for Hybrid Speech Recognition (2019, methodology): “Self-attention and transformer models were also used for ASR, mostly in the sequence-to-sequence architecture [17–19] with notable exceptions of [20,21].”
Efficient Attention: Attention with Linear Complexities (2018, background): “s highly successful on sequence tasks. They hold the state-ofthe-art records on virtually all tasks in natural language processing [6,21,28] and is highly competitive on end-to-end speech recognition [7,19]. [26] ﬁrst adapted dot-product attention for computer vision and proposed the non-local modul”
Understanding and Overcoming the Challenges of Efficient Transformer Quantization (2021, background): “They are also increasingly adopted in other areas, including vision (Carion et al., 2020; Dosovitskiy et al., 2020; Chen et al., 2020; Girdhar et al., 2019) and audio (Dong et al., 2018; Child et al., 2019).”
WENETSPEECH: A 10000+ Hours Multi-Domain Mandarin Corpus for Speech Recognition (2021, background): “On the one hand, the development of neural networks has increased the capacity of models, pushing the dominant framework from the hybrid hidden Markov models [1, 2] to end-to-end models, like CTC [3, 4], RNNT [5, 6], and encoder-decoder based models [7, 8, 9, 10, 11].”
FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning (2022, background): “Despite the success of deep learning in various fields [1, 2, 3], its superior performance heavily relies on supervised training with sufficient labeled data.”
Recurrent Memory Transformer (2022, background): “Transformers (Vaswani et al., 2017) have been widely adopted across multiple domains and tasks (Radford et al., 2018; Dong et al., 2018; Devlin et al., 2019; Dosovitskiy et al., 2021; Ramesh et al., 2021; Jaegle et al., 2021).”
Combiner: Full Attention Transformer with Sparse Computation Cost (2021, background): “It has also achieved excellent results in other domains like image recognition [9], code understanding [10], speech recognition [11], protein [12], music [13] and image [14] generative modeling.”
A Survey of Transformers (2021, methodology): “Audio Speech Transformer[31], Streaming Transformer[15], Reformer-TTS[57], Music Transformer[56]”
Diagonal State Spaces are as Effective as Structured State Spaces (2022, background): “This recipe has also led to remarkable developments in the areas of vision [RPG ` 21, RKH ` 21] and speech [DXX18, BZMA20].”
Self-attention Networks for Connectionist Temporal Classification in Speech Recognition (2019, methodology): “Unlike past works, we do not require convolutional frontends [19] or interleaved recurrences [27] to train self-attention for ASR.”