PyTorch: An Imperative Style, High-Performance Deep Learning Library

By Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga and 11 more. 2019

53,912 citations recorded by the source corpus.

Deep learning frameworks have often focused on either usability or speed, but not both. PyTorch is a machine learning library that shows that these two goals are in fact compatible: it was designed from first principles to support an imperative and Pythonic programming style that supports code as a model, makes debugging easy and is consistent with other popular scientific computing libraries, while remaining efficient and supporting hardware accelerators such as GPUs. In this paper, we detail the principles that drove the implementation of PyTorch and how they are reflected in its architecture. We emphasize that every aspect of PyTorch is a regular Python program under the full control of its user. We also explain how the careful and pragmatic implementation of the key components of its runtime enables them to work together to achieve compelling performance. We demonstrate the efficiency of individual subsystems, as well as the overall speed of PyTorch on several commonly used benchmarks.

Subfields: Artificial Intelligence, Hardware and Architecture, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Yoshua Bengio
  3. Jianfeng Gao
  4. Percy Liang
  5. Quoc V. Le
  6. Luke S. Zettlemoyer
  7. Sergey Levine
  8. Ruslan Salakhutdinov

In Hardware and Architecture

  1. Quoc V. Le
  2. Zi-Mei Yao
  3. Kurt Keutzer
  4. Yuxiong He
  5. Matei A. Zaharia
  6. Ali Ayub
  7. Tri Dao
  8. Alan R. Wagner

In ML Systems & Hardware

  1. De-chuan Zhan
  2. Da-Wei Zhou
  3. Martin Jaggi
  4. Sebastian Raschka
  5. Yoshua Bengio
  6. Michael J. Black
  7. Ali Ayub
  8. Han-Jia Ye

Citations in the corpus