Extracting deep neural network bottleneck features using low-rank matrix factorization

By Yu Zhang, Ekapol Chuangsuwanich, James R. Glass. 2014

71 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Efficiency & Compression, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tara N. Sainath
2. Bo Li
3. Jinyu Li
4. Yoshua Bengio
5. Dong Yu
6. Ron J. Weiss
7. Arun Narayanan
8. Yu Zhang

In Representation & Self-Supervised Learning

1. Zeyuan Allen-Zhu
2. Yuanzhi Li
3. Fred Richardson
4. Najim Dehak
5. Douglas A. Reynolds
6. Kevin Wilson
7. Bo Li
8. Yelong Shen

In Efficiency & Compression

1. Tara N. Sainath
2. Ron J. Weiss
3. Bo Li
4. Zeyuan Allen-Zhu
5. Fred Richardson
6. Weizhu Chen
7. Yuanzhi Li
8. Jia-Nan Wu

In Signal Processing

1. Tara N. Sainath
2. Bo Li
3. Jinyu Li
4. Ron J. Weiss
5. Arun Narayanan
6. Dong Yu
7. Michiel Bacchiani
8. Yu Zhang

Citations in the corpus

• LoRA: Low-Rank Adaptation of Large Language Models (2021, background): “Some prior works even explicitly impose the low-rank constraint when training the original neural network, such as [17, 30, 35, 38, 39], however, to the best of our knowledge, none of these works considers low-rank update for adaptation to downstream tasks.”
• Compression of fully-connected layer in neural network by Kronecker product (2015, background): “Zhang et al. (2014a) considered approximation of multiple layers with nonlinear activations.”
• Neural Network Adaptive Beamforming for Robust Multichannel Speech Recognition (2016, methodology): “Since the softmax prediction is very high dimensional, we feed back the low-rank activations preceding the softmax to the FP module to limit the increase of model parameters [25].”
• Deep Neural Network Approaches to Speaker and Language Recognition (2015, methodology): “The first indirect method (bottleneck features or BNFs) uses frame-level features extracted from a DNN with a special bottleneck layer [15] and the second indirect method (DNN posteriors) uses posteriors extracted from a DNN to accumulate multi-modal statistics [6].”
• A unified deep neural network for speaker and language recognition (2015, methodology): “The first indirect method (bottleneck features or BNFs) uses frame-level features extracted from a DNN with a special bottleneck layer [15] and the second indirect method (DNN posteriors) uses posteriors extracted from a DNN to accumulate multi-modal statistics [6].”
• Multichannel Signal Processing With Deep Neural Networks for Automatic Speech Recognition (2017, methodology): “Since the softmax prediction is very high dimensional, we feed back the low-rank activations preceding the softmax to the FP module to limit the increase of model parameters [36].”