An introduction to computational networks and the computational network toolkit (invited talk)

By Dong Yu, Adam Eversole, M. Seltzer, Kaisheng Yao, B. Guenter, Oleksii Kuchaiev, Frank Torsten Bernd Seide, Huaming Wang, Jasha Droppo, Zhiheng Huang and 4 more. 2014

402 citations recorded by the source corpus.

Subfields: Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

Citations in the corpus

Training Deep Nets with Sublinear Memory Cost (2016, background): “Theano [5, 4] is a pioneering framework to bring the computation graph to deep learning, which is joined by recently introduced frameworks such as CNTK [2], Tensorflow [1] and MXNet [6].”
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems (2016, neutral): “Theano [7], Torch [13], Caffe [26], Chainer [49] and the Computational Network Toolkit [54] are a few systems designed primarily for the training of neural networks.”
AllenNLP: A Deep Semantic Natural Language Processing Platform (2018, background): “Most existing deep-learning toolkits are designed for general machine learning (Bergstra et al., 2010; Yu et al., 2014; Chen et al., 2015; Abadi et al., 2016; Neubig et al., 2017), and can require signiﬁ-cant effort to develop research infrastructure for particular model classes.”
QSGD: Communication-Optimal Stochastic Gradient Descent, with Applications to Training Neural Networks (2016, background): “Of note, we show that there exist generic parameter settings under which all known network architectures preserve or slightly improve their full accuracy when using quantization.”
Residual LSTM: Design of a Deep Recurrent Architecture for Distant Speech Recognition (2017, methodology): “The computational network toolkit (CNTK) [26] is used to train and decode three acoustic models.”
Sequence-to-sequence neural net models for grapheme-to-phoneme conversion (2015, methodology): “We plan to release our training recipes to public through computation network toolkit (CNTK) [31].”
Achieving Human Parity in Conversational Speech Recognition (2016, methodology): “In CNTK, a neural network (and the training criteria) are specified by its formula, using a custom functional language (BrainScript), or Python.”
The Convergence of Sparsified Gradient Methods (2018, background): “, [1, 31, 5]; due to space limits, we focus on communication-reduction techniques that are closely related.”
The microsoft 2016 conversational speech recognition system (2016, methodology): “h, and dynamically applies the language model score. The unigram graph has about 300k states and 500k arcs. All acoustic models were trained using the open-source Computational Network Toolkit (CNTK) [40]. Table 4 shows the result of i-vector adaptation and LFMMI training on several of our systems. ”
Multitalker Speech Separation With Utterance-Level Permutation Invariant Training of Deep Recurrent Neural Networks (2017, methodology): “ames. This makes uPIT a simple and attractive solution. VI. EXPERIMENTAL RESULTS We evaluated uPIT on various setups and all models were implemented using the Microsoft Cognitive Toolkit (CNTK) [47], [48]2. The models were evaluated on their potential to improve the Signal-to-Distortion Ratio (SDR) ”
OpenSeq2Seq: Extensible Toolkit for Distributed and Mixed Precision Training of Sequence-to-Sequence Models (2018, methodology): “However, implementing a sequence-to-sequence model in a general purpose deep learning framework such as TensorFlow (Abadi et al., 2016), CNTK (Yu et al., 2014) or PyTorch (Paszke et al., 2017) can be challenging, especially with support for distributed training.”
Highway long short-term memory RNNS for distant speech recognition (2015, methodology): “Index Terms— Highway LSTM, CNTK, LSTM, Sequence Training”
Learning to Match using Local and Distributed Representations of Text for Web Search (2016, methodology): “We implemented our model using CNTK [41] and trained the model with stochastic gradient descent based optimization (with automatic differentiation) on a single GPU.”
Extracting Domain Invariant Features by Unsupervised Learning for Robust Automatic Speech Recognition (2018, methodology): “The Computational Network Toolkit (CNTK) [22] is used for neural network-based acoustic model training.”
Permutation invariant training of deep models for speaker-independent multi-talker speech separation (2016, methodology): “Our models were implemented using the Microsoft Cognitive Toolkit (CNTK) [28].”
Speech Enhancement in Multiple-Noise Conditions Using Deep Neural Networks (2016, methodology): “CNTK[36] is used for all of our experiments.”
Learning to Optimize Tensor Programs (2018, background): “Scalable learning systems [1, 4, 8, 2] rely on manually optimized, high-performance tensor operation libraries, such as cuDNN, that are optimized for a narrow range of hardware devices.”
Tiresias: A GPU Cluster Manager for Distributed Deep Learning (2019, methodology): “Model and data partitions are determined by the DL framework and/or the user [9, 16, 42].”
Factorized Hidden Layer Adaptation for Deep Neural Network Based Acoustic Modeling (2016, methodology): “We use CNTK [71] to train the DNNs.”