Low-rank plus diagonal adaptation for deep neural networks

By Yong Zhao, Jinyu Li, Yifan Gong. 2016

71 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Efficiency & Compression, Optimization & Training, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Efficiency & Compression

In Optimization & Training

In Signal Processing

Citations in the corpus

LoRA: Low-Rank Adaptation of Large Language Models (2021, background): “Some prior works even explicitly impose the low-rank constraint when training the original neural network, such as [17, 30, 35, 38, 39], however, to the best of our knowledge, none of these works considers low-rank update for adaptation to downstream tasks.”
Linear Networks Based Speaker Adaptation for Speech Synthesis (2018, methodology): “In paper [22], the diagonal matrix Dk×k is initialized with identity matrix, which is fine-tuned or fixed during adaptation stage.”
Unsupervised adaptation with domain separation networks for robust speech recognition (2017, background): “In [15, 16, 17], the trainable parameters are further reduced by singular value decomposition of weight matrices of a neural network.”
Conditional Teacher-student Learning (2019, unknown): “[30, 31], subspace-based [32, 33] and adversarial learning-based [34, 35] approaches.”
Factorized Hidden Layer Adaptation for Deep Neural Network Based Acoustic Modeling (2016, methodology): “Furthermore, some approaches applied singular value decomposition (SVD) pruning [37] and followed by a similar linear layer based adaptation in a low dimensional space [38]–[41] which reduces the number of adaptation parameters.”
Domain Adaptation Using Factorized Hidden Layer for Robust Automatic Speech Recognition (2018, background): “…i-vector feature augmentation [19, 20], speaker-code adaptation [21], cluster adaptive training (CAT) [22, 23], factorized hidden layer [9], learning hidden unit contribution (LHUC) [24], singular value decomposition based subspace adaptation [25] and low rank plus diagonal (LRPD) [26, 27].”
Speaker-Invariant Training Via Adversarial Learning (2018, background): “Therefore, speaker adaptation meth-ods are widely used to boost the recognition system performance [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13].”
An Investigation Into On-device Personalization of End-to-end Automatic Speech Recognition Models (2019, unknown): “A common solution to this problem is to adapt a well-trained model using a small amount of user-specific data to build a personalized model for each user [4, 5, 6, 7, 8, 9, 10, 11, 12, 13].”