TensorFlow Eager: A Multi-Stage, Python-Embedded DSL for Machine Learning

By Akshay Agrawal, A. Modi, Alexandre Passos, Allen Lavoie, Ashish Agarwal, Asim Shankar, Igor Ganichev, J. Levenberg, Mingsheng Hong, Rajat Monga and 1 more. 2019

92 citations recorded by the source corpus.

TensorFlow Eager is a multi-stage, Python-embedded domain-specific language for hardware-accelerated machine learning, suitable for both interactive research and production. TensorFlow, which TensorFlow Eager extends, requires users to represent computations as dataflow graphs; this permits compiler optimizations and simplifies deployment but hinders rapid prototyping and run-time dynamism. TensorFlow Eager eliminates these usability costs without sacrificing the benefits furnished by graphs: It provides an imperative front-end to TensorFlow that executes operations immediately and a JIT tracer that translates Python functions composed of TensorFlow operations into executable dataflow graphs. TensorFlow Eager thus offers a multi-stage programming model that makes it easy to interpolate between imperative and staged execution in a single package.

Subfields: Artificial Intelligence, Hardware and Architecture, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Quoc V. Le
2. Sergey Levine
3. J. Kolter
4. Yonghui Wu
5. Yoshua Bengio
6. Mohit Bansal
7. Mohammad Rastegari
8. Ruoming Pang

In Hardware and Architecture

1. J. Kolter
2. Brandon Amos
3. Steven Diamond
4. Sanjay Ghemawat
5. Stephen P. Boyd
6. Michael Isard
7. Xuehai Qian
8. Myeongjae Jeon

In ML Systems & Hardware

1. J. Kolter
2. Brandon Amos
3. Sanjay Ghemawat
4. Stephen P. Boyd
5. Michael Isard
6. Mohammad Rastegari
7. Xuehai Qian
8. Aakanksha Chowdhery

Citations in the corpus

• Pathways: Asynchronous Distributed Dataflow for ML (2022, methodology): “Examples of this architecture include MPI (Clarke et al., 1994), Py-Torch (Paszke et al., 2019), JAX (Bradbury et al., 2018), and more recent configurations of TensorFlow (Shazeer et al., 2018; Agrawal et al., 2019).”
• The Deep Learning Compiler: A Comprehensive Survey (2020, background): “With the continuous efforts from both industry and academia, several popular DL frameworks have been proposed such as TensorFlow [10], PyTorch [11], MXNet [12] and CNTK [13], in order to simplify the implementation of various DL models.”
• Differentiable Convex Optimization Layers (2019, methodology): “We also implement differentiable convex optimization layers in PyTorch [66] and TensorFlow 2.0 [2].”
• Zico: Efficient GPU Memory Sharing for Concurrent DNN Training (2021, background): “, classification based on tensor access time intervals in eager mode [2].”
• DKM: Differentiable K-Means Clustering Layer for Neural Network Compression (2021, methodology): “The iterative process will be dynamically executed in an imperative execution in PyTorch [14] and Tensorflow-Eager [1] and is differentiable for backward-propagation, as W̃ is based on the attention between weights and centroids.”
• Capuchin: Tensor-based GPU Memory Management for Deep Learning (2020, background): “, Pytorch [23] and Tensorfow’s eager mode [3].”
• Nimble: Lightweight and Parallel GPU Task Scheduling for Deep Learning (2020, methodology): “Second, DL frameworks including PyTorch and TensorFlow Eager describe a neural network as an imperative Python program.”
• CrypTen: Secure Multi-Party Computation Meets Machine Learning (2021, background): “development cycle, make debugging harder, and prevent users from using arbitrary host-language constructs [3].”