Semi-Orthogonal Low-Rank Matrix Factorization for Deep Neural Networks

By Daniel Povey, Gaofeng Cheng, Yiming Wang, Ke Li, Hainan Xu, Mahsa Arab Yarmohammadi, Sanjeev Khudanpur. 2018

522 citations recorded by the source corpus.

Time Delay Neural Networks (TDNNs), also known as one-dimensional Convolutional Neural Networks (1-d CNNs), are an efﬁcient and well-performing neural network architecture for speech recognition. We introduce a factored form of TDNNs (TDNN-F) which is structurally the same as a TDNN whose layers have been compressed via SVD, but is trained from a random start with one of the two factors of each matrix constrained to be semi-orthogonal. This gives substantial improvements over TDNNs and performs about as well as TDNN-LSTM hybrids.

Subfields: Artificial Intelligence, Model Architectures, Optimization & Training, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Optimization & Training

In Signal Processing

Citations in the corpus

LoRA: Low-Rank Adaptation of Large Language Models (2021, background): “Some prior works even explicitly impose the low-rank constraint when training the original neural network, such as [17, 30, 35, 38, 39], however, to the best of our knowledge, none of these works considers low-rank update for adaptation to downstream tasks.”
CHiME-6 Challenge: Tackling Multispeaker Speech Recognition for Unsegmented Recordings (2020, positive): “Chain model training (stage 15) We use a factorized time delay neural network (TDNN-F) adapted from the Switchboard recipe 7q model [38].”
TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation (2018, methodology): “We present experimental results with diﬀerent ASR architectures, by using Time Delay Neural Network (TDNN) [5] and Factored TDNN (TDNN-F) acoustic models [7] on the legacy distribution of TED-LIUM 3 in section 3, and also exploring the use of a pure neural end-to-end system in section 4.”
Jasper: An End-to-End Convolutional Neural Acoustic Model (2019, methodology): “Traditional N-gram LMs have been augmented with neural LMs in recent work [21, 22, 23].”
WENETSPEECH: A 10000+ Hours Multi-Domain Mandarin Corpus for Speech Recognition (2021, methodology): “Then, we train a chain model, which stacks 6 convolutional neural network (CNN) layers, 9 factored time-delay neural network (TDNN-F) layers [30], 1 time-restricted attention layer [31] (H = 30), 2 projected long short-term memory (LSTMP) with TDNN-F blocks.”
From Senones to Chenones: Tied Context-Dependent Graphemes for Hybrid Speech Recognition (2019, methodology): “DNN is subsequently replaced with variants of Recurrent Neural Network (RNN) and Convolutional Neural Network (CNN) [5–10], whose primary advantage comes from their ability to model long temporal context.”
Automatic Lyrics Transcription in Polyphonic Music: Does Background Music Help? (2019, methodology): “A factorized time-delay neural network (TDNN-F) model [39] with additional convolutional layers (2 convolutional, 10 time-delay layers followed by a rank reduction layer) was trained according to the standard Kaldi recipe (version 5.4).”
ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification (2020, background): “Another, complementary way to exploit multi-layer information is to use the output of all preceding SE-Res2Blocks and initial convolutional layer as input for each frame layer block [15, 17].”
DiPCo - Dinner Party Corpus (2019, unknown): “The AM was a Time Delay Neural Network and Factored (TDNN-F) deep neural network [4] with 15 layers, the dimension of each layer was 1536 and the bottleneck dimension was 160.”
An Investigation into the Effectiveness of Enhancement in ASR Training and Test for Chime-5 Dinner Party Transcription (2019, background): “However, recently it has been shown that introducing factorized layers into the TDNN architecture facilitates training deeper networks and also improves the ASR performance [25].”
Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Tasks (2019, background): “With the ability to learn rich distributed representations of data in an end-to-end fashion, deep neural networks have achieved the state of the arts in a variety of ﬁelds (He et al., 2017; Vaswani et al., 2017; Povey et al., 2018; Yu et al., 2018).”
ShortGPT: Layers in Large Language Models are More Redundant Than You Expect (2024, methodology): “(2018) is a model compression technique aimed at approximating a given weight matrix by decomposing it into two or more smaller matrices with significantly lower dimensions.”
The STC System for the CHiME-6 Challenge (2020, methodology): “As in [19], this enhancement was also applied for training data augmentation and provided a signiﬁcant WER reduction.”
A comprehensive survey on model compression and acceleration (2020, methodology): “Furthermore, Povey et al. (2018) represented a factored one-dimensional CNN, like time delay NN (Peddinti et al. 2015) (1-dimensional CNNs) in which the network layers were compressed using SVD but, trained by considering one of the factored matrices constrained to be semi-orthogonal.”
State-of-the-Art Speech Recognition Using Multi-Stream Self-Attention with Dilated 1D Convolutions (2019, methodology): “In order to make the convolution layers more efﬁcient, we utilize the factorized TDNN [17].”
Acoustic Modeling for Overlapping Speech Recognition: Jhu Chime-5 Challenge System (2019, methodology): “TDNN-F architecture is a factored form of TDNN introduced in [23].”