Automatic differentiation in PyTorch

By Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, E. Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, Adam Lerer. 2017

16,222 citations recorded by the source corpus.

Subfields: ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In ML Systems & Hardware

Citations in the corpus

RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019, methodology): “We release our model, pretraining and ﬁne-tuning code implemented in PyTorch (Paszke et al., 2017).”
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference (2020, methodology): “Our implementation is based on the Transformers library (Wolf et al., 2019) and Py-Torch (Paszke et al., 2017).”
A Constructive Prediction of the Generalization Error Across Scales (2019, methodology): “For ImageNet we use ResNet-50 and build on the code from the PyTorch framework (Paszke et al., 2017) to vary the model width.”
UNITER: Learning UNiversal Image-TExt Representations (2019, neutral): “(Paszke et al., 2017).”
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (2019, methodology): “• To facilitate the development of unified new methods for this more diverse set of tasks, SuperGLUE is distributed with a modular modeling toolkit for work on pretraining, multitask learning, and transfer learning in NLP, built on PyTorch (Paszke et al., 2017) and AllenNLP (Gardner et al.”
Cross-lingual Language Model Pretraining (2019, methodology): “We implement all our models in Py-Torch (Paszke et al., 2017), and train them on 64 Volta GPUs for the language modeling tasks, and 8 GPUs for the MT tasks.”
Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks (2018, methodology): “To implement intermediate-task and target-task training, we use the jiant transfer learning toolkit, 3 which is built on AllenNLP (Gardner et al. , 2017) and PyTorch (Paszke et al., 2017).”
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering (2018, methodology): “B Implementation and Training Our neural models are implemented with AllenNLP15 (Gardner et al., 2017) and PyTorch16 (Paszke et al., 2017).”
Unifying Question Answering and Text Classification via Span Extraction (2019, methodology): “We primarily rely on the BERT training library 1 available in PyTorch (Paszke et al., 2017).”
Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning (2018, methodology): “We are also grateful to the PyTorch development team (Paszke et al., 2017).”
Mixed Precision Training (2017, methodology): “baseline FP32 training session using identical hyper-parameters. Networks were trained using Caffe (Jia et al., 2014) framework modiﬁed to use Volta TensorOps, except for Resnet50 which used PyTorch (Paszke et al., 2017). 5 Published as a conference paper at ICLR 2018 Training schedules were used fr”
Revisiting LSTM Networks for Semi-Supervised Text Classification via Mixed Objective Function (2019, methodology): “All of our models were implemented in PyTorch framework (Paszke et al. 2017) and were trained on a single GPU.”
Gradient Descent Happens in a Tiny Subspace (2018, methodology): “This product can be computed in common autograd packages such as Tensor-Flow (Abadi et al.) or PyTorch (Paszke et al., 2017) as follows.”
Efficient Large Scale Language Modeling with Mixtures of Experts (2021, methodology): “We train our models in PyTorch (Paszke et al., 2017) using FAIRSEQ (Ott et al.”
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020, methodology): “Software Software systems supporting neural networks evolved together with the advancement of the underlying hardware (Dean et al., 2012; Bastien et al., 2012; Abadi et al., 2016; Paszke et al., 2017; Palkar & Zaharia, 2019).”
Root Mean Square Layer Normalization (2019, methodology): “satisfactory convergence with a partial ratio of 6.25%. 6 Experiments To test the efﬁciency of layer normalization across different implementations, we perform experiments with Tensorﬂow [1], PyTorch [20] and Theano [29]. We add RMSNorm to different models, comparing against an unnormalized baseline”
HuggingFace's Transformers: State-of-the-art Natural Language Processing (2019, positive): “The second direction concerns lower-level deep-learning frameworks like PyTorch (Paszke et al., 2017) and TensorFlow (Abadi et al., 2015) which have both been extended with model sharing capabilities or hubs, respectively called TensorFlow Hub 18 and PyTorch Hub 19 .”
Finding Generalizable Evidence by Learning to Convince Q&A Models (2019, methodology): “We build off AllenNLP (Gardner et al., 2018) using PyTorch (Paszke et al., 2017).”
Large Memory Layers with Product Keys (2019, methodology): “We implement our models with PyTorch [33], and train them on 32 Volta GPUs.”
Knowledge Guided Text Retrieval and Reading for Open Domain Question Answering (2019, methodology): “All experiments are done in Python 3.5 and PyTorch 1.1.0 (Paszke et al., 2017).”