DyNet: The Dynamic Neural Network Toolkit

By Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Bridger Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, David Chiang, Daniel Clothiaux, Trevor Cohn and 15 more. 2017

391 citations recorded by the source corpus.

We describe DyNet, a toolkit for implementing neural network models based on dynamic declaration of network structure. In the static declaration strategy that is used in toolkits like Theano, CNTK, and TensorFlow, the user first defines a computation graph (a symbolic representation of the computation), and then examples are fed into an engine that executes this computation and computes its derivatives. In DyNet's dynamic declaration strategy, computation graph construction is mostly transparent, being implicitly constructed by executing procedural code that computes the network outputs, and the user is free to use different network structures for each input. Dynamic declaration thus facilitates the implementation of more complicated network architectures, and DyNet is specifically designed to allow users to implement their models in a way that is idiomatic in their preferred programming language (C++ or Python). One challenge with dynamic declaration is that because the symbolic computation graph is defined anew for every training example, its construction must have low overhead. To achieve this, DyNet has an optimized C++ backend and lightweight graph representation. Experiments show that DyNet's speeds are faster than or comparable with static declaration toolkits, and significantly faster than Chainer, another dynamic declaration toolkit. DyNet is released open-source under the Apache 2.0 license and available at this http URL.

Subfields: Artificial Intelligence, ML Systems & Hardware, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In ML Systems & Hardware

In Signal Processing

Citations in the corpus

PyTorch: An Imperative Style, High-Performance Deep Learning Library (2019, methodology): “This design was pioneered for model authoring by Chainer[5] and Dynet[7].”
AllenNLP: A Deep Semantic Natural Language Processing Platform (2018, background): “Most existing deep-learning toolkits are designed for general machine learning (Bergstra et al., 2010; Yu et al., 2014; Chen et al., 2015; Abadi et al., 2016; Neubig et al., 2017), and can require signiﬁ-cant effort to develop research infrastructure for particular model classes.”
Latent Multi-Task Architecture Learning (2017, methodology): “We implement all models in DyNet (Neubig et al. 2017) and make our code available at https://github.com/ sebastianruder/sluice-networks .”
Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow (2018, methodology): “6 We implement our neural correspondencemodel using theDyNet neural network toolkit [26].”
Abstract Syntax Networks for Code Generation and Semantic Parsing (2017, methodology): “Our system is implemented in Python using the DyNet neural network library (Neubig et al., 2017).”
Marian: Fast Neural Machine Translation in C++ (2018, methodology): “The model architecture in Sennrich et al. (2017a) is a sequence-to-sequence model with single-layer RNNs in both, the encoder and decoder. The RNN in the encoder is bi-directional. Depth is achieved by building stacked GRU-blocks resulting in very tall RNN cells for every recurrent step (deep transi”
Neural Semantic Parsing with Type Constraints for Semi-Structured Tables (2017, methodology): “This Scala library combines ideas from dynamic neural network frameworks (Neubig et al., 2017) and probabilistic programming (Goodman and Stuhlmüller, 2014) to simplify the implementation of complex neural structured prediction models.”
Dynamic Entity Representations in Neural Language Models (2017, methodology): “Our model is implemented with DyNet (Neubig et al., 2017) and available at https://github.com/jiyfeng/entitynlm .”
Learning Language Representations for Typology Prediction (2017, methodology): “The implementation of both the LM and NMT models described in §3 was done in the DyNet toolkit (Neubig et al., 2017).”
Stronger Baselines for Trustable Results in Neural Machine Translation (2017, methodology): “Our starting point for experimentation is a standard baseline neural machine translation system implemented using the Lamtram 1 and DyNet toolkits (Neubig, 2015; Neubig et al., 2017).”
Extreme Adaptation for Personalized Neural Machine Translation (2018, methodology): “5 All our models are implemented with the DyNet (Neubig et al., 2017) framework, and unless speci-ﬁed we use the default settings therein.”
Strong Baselines for Neural Semi-Supervised Learning under Domain Shift (2018, neutral): “We implement our models in DyNet (Neubig et al., 2017).”
Mimicking Word Embeddings using Subword RNNs (2017, methodology): “We initialize character embeddings randomly, and use DyNet to implement the model (Neubig et al., 2017).”
SparseMAP: Differentiable Sparse Structured Inference (2018, neutral): “0.2 (Neubig et al., 2017).”
Improving Text-to-SQL Evaluation Methodology (2018, positive): “3 presents the overall architecture, which we implemented in DyNet (Neubig et al., 2017).”
A hybrid method of exponential smoothing and recurrent neural networks for time series forecasting (2020, positive): “The method was written in C++ relying on the DyNet library (Neubig et al., 2017).”
Analyzing Assumptions in Conversation Disentanglement Research Through the Lens of a New Dataset and Model (2018, methodology): “We explored a range of models, all implemented using DyNet (Neubig et al., 2017).”