PyTorch: An Imperative Style, High-Performance Deep Learning Library
By Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga and 11 more. 2019
53,912 citations recorded by the source corpus.
Deep learning frameworks have often focused on either usability or speed, but not both. PyTorch is a machine learning library that shows that these two goals are in fact compatible: it was designed from first principles to support an imperative and Pythonic programming style that supports code as a model, makes debugging easy and is consistent with other popular scientific computing libraries, while remaining efficient and supporting hardware accelerators such as GPUs. In this paper, we detail the principles that drove the implementation of PyTorch and how they are reflected in its architecture. We emphasize that every aspect of PyTorch is a regular Python program under the full control of its user. We also explain how the careful and pragmatic implementation of the key components of its runtime enables them to work together to achieve compelling performance. We demonstrate the efficiency of individual subsystems, as well as the overall speed of PyTorch on several commonly used benchmarks.
Subfields: Artificial Intelligence, Hardware and Architecture, ML Systems & Hardware
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Yoshua Bengio
- Jianfeng Gao
- Percy Liang
- Quoc V. Le
- Luke S. Zettlemoyer
- Sergey Levine
- Ruslan Salakhutdinov
In Hardware and Architecture
In ML Systems & Hardware
- De-chuan Zhan
- Da-Wei Zhou
- Martin Jaggi
- Sebastian Raschka
- Yoshua Bengio
- Michael J. Black
- Ali Ayub
- Han-Jia Ye
Citations in the corpus
- End-to-End Object Detection with Transformers (2020, methodology): “Inference code for DETR can be implemented in less than 50 lines in PyTorch [32].”
- Scaling Up Models and Data with t5x and seqio (2022, methodology): “We therefore additionally introduce seqio , a fully-featured library for efficiently managing data pipelines and model evaluation with a simple, task-based API. seqio builds off of tensorflow.data with additional support for SPMD-based data parallelism, making it compatible with many modeling framework”
- Pathways: Asynchronous Distributed Dataflow for ML (2022, background): “Examples of this architecture include MPI (Clarke et al., 1994), Py-Torch (Paszke et al., 2019), JAX (Bradbury et al., 2018), and more recent configurations of TensorFlow (Shazeer et al., 2018; Agrawal et al., 2019).”
- GSPMD: General and Scalable Parallelization for ML Computation Graphs (2021, methodology): “Multiple front-end frameworks including TensorFlow [5], JAX [7], PyTorch [29] and Julia [6] already have lowering logic to transform their graph representation to XLA HLO graph, which can then be compiled into target executables if the accelerator has a compiler backend for XLA.”
- Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM (2021, positive): “Second, we generated fused kernels for a sequence of element-wise operations (bias + GeLU and bias + dropout + add) using PyTorch JIT [10].”
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022, background): “[62] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.”
- InCoder: A Generative Model for Code Infilling and Synthesis (2022, methodology): “For our learning rate scheduler, we use the built-in polynomial decay learning rate scheduler available in [49] with 1500 warmup updates.”
- Efficient Memory Management for Large Language Model Serving with PagedAttention (2023, positive): “For the model executor, we implement popular LLMs such as GPT [5], OPT [62], and LLaMA [52] PyTorch [39] and Transformers [58].”
- MegaBlocks: Efficient Sparse Training with Mixture-of-Experts (2022, methodology): “We implemented our techniques in a system called MegaBlocks, which builds on Megatron-LM (Shoeybi et al., 2019) and PyTorch (Paszke et al., 2019).”
- End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering (2021, methodology): “We use PyTorch (Paszke et al., 2019) to implement our proposed model and relevant baselines.”
- Improving and Simplifying Pattern Exploiting Training (2021, methodology): “Our code is implemented in Pytorch (Paszke et al., 2019) using HuggingFace (Wolf et al., 2020).”
- Pretrained Transformers for Text Ranking: BERT and Beyond (2020, neutral): “ovation. Today, the Transformers library75 by Hugging Face [Wolf et al., 2019] has emerged as the de facto standard implementation of BERT as well as many transformer models, supporting both PyTorch [Paszke et al., 2019] and TensorFlow [Abadi et al., 2016], the two most popular deep learning librari”
- Visual Transformers: Token-based Image Representation and Processing for Computer Vision (2020, positive): “We implement VT models in PyTorch [29].”
- Learning Visual Representations with Caption Annotations (2020, methodology): “We use PyTorch [45] and the mixed-precision functionality provided by NVIDIA Apex 2 to perform all experiments.”
- Array programming with NumPy (2020, methodology): “For example, each deep learning framework created its own arrays; PyTorch [38], Tensorflow [39], Apache MXNet [40], and JAX arrays all have the capability to run on CPUs and GPUs, in a distributed fashion, utilizing lazy evaluation to allow for additional performance optimizations.”