CNTK: Microsoft's Open-Source Deep-Learning Toolkit

By Frank Torsten Bernd Seide, Amit Agarwal. 2016

449 citations recorded by the source corpus.

Subfields: Artificial Intelligence, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In ML Systems & Hardware

Citations in the corpus

PipeDream: Fast and Efficient Pipeline Parallel DNN Training (2018, methodology): “CNTK’s [33] 1-bit quantization technique tackles the problem of communication bottlenecks in data parallelism training [35].”
Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization (2019, background): “In this paper, we are particularly interested in synchronous distributed stochastic gradient descent algorithms for non-convex optimization problems mainly due to their recent successes and popularity in deep learning models [26, 29, 46, 47].”
$\texttt{DeepSqueeze}$: Decentralization Meets Error-Compensated Compression (2019, methodology): “For example, the parameter server [Abadi et al., 2016, Li et al., 2014], AllReduce [Renggli et al., 2018, Seide and Agarwal, 2016], adaptive distributed learning, diﬀerentially private distributed learning, distributed proximal primal-dual algorithm, non-smooth distributed optimization,…”
On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization (2019, methodology): “We run both Algorithm 1 and the model averaging with Polyak’s momentum SGD suggested in (Seide & Agarwal, 2016; Wang & Joshi, 2018b), referred as “model averaging with cleared momentum” to train ResNet56 over CIFAR10.”
Distributed Learning over Unreliable Networks (2018, methodology): “This is a realistic model of running an AllReduce operator implemented with Reduce-Scatter/All-Gather on unreliable network.”
Don't Use Large Mini-Batches, Use Local SGD (2018, background): “Current state-of-the-art distributed deep learning frameworks [18, 19, 20] resort to synchronized large-batch training, allowing scaling by adding more computational units and performing dataparallel synchronous SGD with mini-batches divided between devices.”
Local SGD Converges Fast and Communicates Little (2018, methodology): “This scheme is implemented in state-of-the-art distributed deep learning frameworks [1, 26, 31].”
D2: Decentralized Training over Decentralized Data (2018, methodology): “C-PSGD has been applied to many deep learning frameworks, such as such as CNTK [Seide and Agarwal, 2016], MXNet [Chen et al., 2015], and TensorFlow [Abadi et al., 2016].”
Asynchronous Decentralized Parallel Stochastic Gradient Descent (2017, background): “, 2015), and CNTK (Seide and Agarwal, 2016) support two communication modes: (1) synchronous communication via parameter servers or AllReduce, or (2) asynchronous communication via parameter servers.”
Generalized Byzantine-tolerant SGD (2018, background): “For instance, Tensorﬂow (Abadi et al., 2016), CNTK (Seide & Agarwal, 2016), and MXNet (Chen et al., 2015) implement internal PS’s.”
Forecasting Across Time Series Databases using Long Short-Term Memory Networks on Groups of Similar Series (2017, methodology): “ular variants that allows LSTM gates to examine the state of their memory cell, before updating their states. In this study, we use the Microsoft Cognitive Toolkit (CNTK), an open-source NN tool kit (Seide and Agarwal, 2016), to implement the LSTM. As our base learning algorithm, we use an LSTM with”
Communication Compression for Decentralized Training (2018, methodology): “The centralized algorithms is a widely used scheme for parallel computation, such as Tensorﬂow [Abadi et al., 2016], MXNet [Chen et al., 2015], and CNTK [Seide and Agarwal, 2016].”
Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGD (2018, methodology): “To combine momentum and periodic averaging, we use the block momentum scheme that was proposed in [3, 21] and applied to speech recognition tasks.”
Automatic differentiation in machine learning: a survey (2015, methodology): “, 2014), and CNTK (Seide and Agarwal, 2016) the user first constructs a model as a computational graph using a domain-specific mini language, which then gets interpreted by the framework during execution.”
Stochastic Distributed Learning with Gradient Quantization and Variance Reduction (2019, methodology): “If every block has the same size d/t , then this gives a quantization operator with Block quantization was e.g. used in (Alistarh et al., 2017) and is also implemented (in a similar fashion) in the CNTK toolkit (Seide & Agarwal, 2016).”