Simplified Self-Attention for Transformer-Based end-to-end Speech Recognition

By Haoneng Luo, Shiliang Zhang, Ming Lei, Lei Xie. 2020

39 citations recorded by the source corpus.

Transformer models have been introduced into end-to-end speech recognition with state-of-the-art performance on various tasks owing to their superiority in modeling long-term dependencies. However, such improvements are usually obtained through the use of very large neural networks. Transformer models mainly include two submodules – position-wise feedforward layers and self-attention (SAN) layers. In this paper, to reduce the model complexity while maintaining good performance, we propose a simplified self-attention (SSAN) layer which employs FSMN memory blocks instead of projection layers to form query and key vectors for transformer-based end-to-end speech recognition. We evaluate the SSAN-based and the conventional SAN-based transformers on the public AISHELL-1, internal 1000-hour and 20,000-hour large-scale Mandarin tasks. Results show that our proposed SSAN-based transformer model can achieve over 20% reduction in model parameters and 6.7% relative CER reduction on the AISHELL-1 task. With impressively 20% parameter reduction, our model shows no loss of recognition performance on the 20,000-hour large-scale task.

Subfields: Artificial Intelligence, Natural Language Processing, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Signal Processing

Citations in the corpus

Rethinking Attention with Performers (2020, background): “…architectures that have become SOTA in several areas of machine learning including natural language processing (NLP) (e.g. speech recognition (Luo et al., 2020)), neural machine translation (NMT) (Chen et al., 2018), document generation/summarization, time series prediction, generative…”
Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth (2021, background): “Since then, attention-based architectures have permeated across data domains machine learning applications, such as in natural language processing [DCLT18, POT16], speech recognition [LZL + 20], and computer vision [RPV + 19, BZV + 19].”
Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers (2020, background): “speech recognition [33]), Neural Machine Translation (NMT) [7], document generation/summarization, time series prediction, generative modeling (e.”
Scatterbrain: Unifying Sparse and Low-rank Attention Approximation (2021, background): “[42] Haoneng Luo, Shiliang Zhang, Ming Lei, and Lei Xie.”