Deep Learning with Dynamic Computation Graphs

By Moshe Looks, Marcello Herreshoff, DeLesley S. Hutchins, Peter Norvig. 2017

139 citations recorded by the source corpus.

Neural networks that compute over graph structures are a natural fit for problems in a variety of domains, including natural language (parse trees) and cheminformatics (molecular graphs). However, since the computation graph has a different shape and size for every input, such networks do not directly support batched training or inference. They are also difficult to implement in popular deep learning libraries, which are based on static data-flow graphs. We introduce a technique called dynamic batching, which not only batches together operations between different input graphs of dissimilar shape, but also between different nodes within a single input graph. The technique allows us to create static graphs, using popular libraries, that emulate dynamic computation graphs of arbitrary shape and size. We further present a high-level library of compositional blocks that simplifies the creation of dynamic graph models. Using the library, we demonstrate concise and batch-wise parallel implementations for a variety of models from the literature.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

Learned in Translation: Contextualized Word Vectors (2017, background): “This is evidence for our hypothesis that MT data has potential as a large resource for transfer learning in NLP. [Looks et al., 2017] 89.4 SVM [Van-Tu and Anh-Cuong, 2016] 95.2 TE-LSTM [Huang et al., 2017] 89.6 DSCNN-P [Zhang et al., 2016] 95.6 NSE [Munkhdalai and Yu, 2016a] 89.7 BCN+Char+CoVe…”
Learning to Generate Reviews and Discovering Sentiment (2017, methodology): “The representation learned by our model achieves 91.8% signiﬁcantly outperforming the state of the art of 90.2% by a 30 model ensemble (Looks et al., 2017).”
Search-based Neural Structured Learning for Sequential Question Answering (2017, background): “Unlike previous dynamic neural network frameworks (Andreas et al., 2016; Looks et al., 2017), where each example can have different but predetermined structure, DynSP needs to dynamically explores and constructs different neural network structures for each question.”
Ray: A Distributed Framework for Emerging AI Applications (2017, background): “TensorFlow Fold [33] provides some support for dynamic task graphs, as well as MXNet through its internal C++ APIs, but neither fully supports the ability to modify the DAG during execution in response to task progress, task completion times, or faults.”
Dynamic-Structured Semantic Propagation Network (2018, methodology): “Instead of using single instance training in most of treestructured [34] and graph-structured [21, 19] networks, our DSSPN uses dynamic graph batching strategy to make good use of efficient data-parallel algorithms and hardware, inspired by very recent attempts [25, 3, 26].”
A Survey of Deep Learning and Its Applications: A New Paradigm to Machine Learning (2019, background): “Looks et al. [32] presented a technique known as dynamic batching which combined the diverse input graphs with dissimilar shapes but with also different nodes that were present in single input graph.”
GamePad: A Learning Environment for Theorem Proving (2018, methodology): “The second optimization involves dynamic batching [12, 17], which enables us to efficiently batch the computation of ops that perform the same operation albeit on different inputs, and thus make better use of the parallel acceleration provided by multi-core CPU’s and GPU’s.”
A computational model for TensorFlow: an introduction (2017, background): “TensorFlow Fold is a library that supports working with dynamic graphs; its design draws on techniques from functional programming such as parser combinators [15].”
Natural Language Interfaces with Fine-Grained User Interaction: A Case Study on Web APIs (2018, methodology): “e Tensorow Fold [22] library is employed to dynamically build the computation graph according to the layout prediction from the controller.”
Ansor : Generating High-Performance Tensor Programs for Deep Learning (2020, background): “The common optimizations at graph level include layout optimizations [32], operator fusion [11, 38, 60], constant folding [42], auto-batching [33], automatic generation of graph substitution [29] and so forth.”
Refining Word Embeddings for Sentiment Analysis (2017, methodology): “…network classifiers that performed well on the Stanford Sentiment Treebank (SST) (Socher et al., 2013) are selected, including convolutional neural networks (CNN) (Kim, 2014), deep averaging network (DAN) (Iyyer et al., 2015) and long-short term memory (LSTM) (Tai et al., 2015; Looks et al., 2017).”
Low latency RNN inference with cellular batching (2018, methodology): “This form of batching is done in TensorFlow Fold [26] and DyNet [30].”
Dynamic control flow in large-scale machine learning (2018, background): “However, for the past few years, we have seen growing demand for dynamic control flow, especially in applications of recurrent and tree neural networks [3, 16, 24] and of reinforcement learning [26].”