TensorFlow: A system for large-scale machine learning
By Martín Abadi, Paul Barham, Jianmin Chen, Z. Chen, Andy Davis, Joseph Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard and 12 more. 2016
19,838 citations recorded by the source corpus.
TensorFlow is a machine learning system that operates at large scale and in heterogeneous environments. Tensor-Flow uses dataflow graphs to represent computation, shared state, and the operations that mutate that state. It maps the nodes of a dataflow graph across many machines in a cluster, and within a machine across multiple computational devices, including multicore CPUs, general-purpose GPUs, and custom-designed ASICs known as Tensor Processing Units (TPUs). This architecture gives flexibility to the application developer: whereas in previous "parameter server" designs the management of shared state is built into the system, TensorFlow enables developers to experiment with novel optimizations and training algorithms. TensorFlow supports a variety of applications, with a focus on training and inference on deep neural networks. Several Google services use TensorFlow in production, we have released it as an open-source project, and it has become widely used for machine learning research. In this paper, we describe the TensorFlow dataflow model and demonstrate the compelling performance that TensorFlow achieves for several real-world applications.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Hardware and Architecture, ML Systems & Hardware
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Quoc V. Le
- Mohit Bansal
- Jianfeng Gao
- Sergey Levine
- Percy Liang
- Ruslan Salakhutdinov
- Luke S. Zettlemoyer
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Bernt Schiele
In Hardware and Architecture
In ML Systems & Hardware
- Suhas N. Diggavi
- Deepesh Data
- Cong Xie
- Oluwasanmi Koyejo
- Indranil Gupta
- Yoshua Bengio
- Ji Liu
- Ce Zhang
Citations in the corpus
- Massive Exploration of Neural Machine Translation Architectures (2017, methodology): “All of the following experiments are run using our own software framework based on Tensor-Flow (Abadi et al., 2016).”
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016, methodology): “The models are trained by a system we implemented using TensorFlow[1].”
- Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “…packages used throughout this project including, but not limited, to Numpy (Harris et al., 2020), SciPy (Virtanen et al., 2020), ftfy (Speer, 2019), TensorFlow (Abadi et al., 2016), PyTorch (Paszke et al., 2019), pandas (pandas development team, 2020), and scikit-learn (Pedregosa et al., 2011).”
- CTRL: A Conditional Transformer Language Model for Controllable Generation (2019, methodology): “This, along with the compressed storage for efficient training (TFRecords) (Abadi et al., 2016), reduces our training data to 140 GB from the total 180 GB collected.”
- Memory-Efficient Adaptive Optimization for Large-Scale Learning (2019, methodology): “We implemented SM3 as an optimizer in TensorFlow [1]; source code is publicly available at [3].”
- Measuring the Effects of Data Parallelism on Neural Network Training (2018, methodology): “Here we present and use the update rules used by the MomentumOptimizer class in TensorFlow (Abadi et al., 2016).”
- Snorkel: Rapid Training Data Creation with Weak Supervision (2017, methodology): “Both models are implemented in Tensorflow [2] and trained using the Adam optimizer [24], with hyperparameters selected via random grid search using a small labeled development set.”
- An Investigation into Neural Net Optimization via Hessian Eigenvalue Density (2019, methodology): “We implemented a large scale version of Algorithm 1 in TensorFlow [1]; the main component is a distributed Lanczos Algorithm.”
- GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021, methodology): “…et al., 2021), including bias in representations (Blodgett et al., 2020), proper handling (Rogers, 2021) and documentation (Bender & Friedman, 2018) of training data, privacy (Abadi et al., 2016b; Carlini et al., 2021), and environmental concerns (Strubell et al., 2019; Patterson et al., 2021).”
- GSPMD: General and Scalable Parallelization for ML Computation Graphs (2021, methodology): “but partitions the graph into pipeline stages . vmap() and TensorFlow’s vectorized_map() .”
- GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020, methodology): “To implement the model in Section 2.1 efficiently on a cluster of devices, we first express the model in terms of linear algebra operations, which are highly tailored and optimized in our software stack TensorFlow (Abadi et al., 2016) and the hardware platform (TPU).”
- Root Mean Square Layer Normalization (2019, methodology): “To test the efficiency of layer normalization across different implementations, we perform experiments with Tensorflow [1], PyTorch [20] and Theano [29].”
- Pretrained Transformers for Text Ranking: BERT and Beyond (2020, neutral): “ary75 by Hugging Face [Wolf et al., 2019] has emerged as the de facto standard implementation of BERT as well as many transformer models, supporting both PyTorch [Paszke et al., 2019] and TensorFlow [Abadi et al., 2016], the two most popular deep learning libraries today. While open source (sharing ”
- Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, methodology): “Our models are trained using TensorFlow [1] on 128 TPU cores for ImageNet and 32 cores for panoptic segmentation.”
- Group Normalization (2018, positive): “GN can be easily implemented by a few lines of code in PyTorch [42] and TensorFlow [1] where automatic differentiation is supported.”