Theano: new features and speed improvements

By Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, James Bergstra, Ian J. Goodfellow, Arnaud Bergeron, Nicolas Bouchard, David Warde-Farley, Yoshua Bengio. 2012

1,415 citations recorded by the source corpus.

Theano is a linear algebra compiler that optimizes a user's symbolically-specified mathematical computations to produce efficient low-level implementations. In this paper, we present new features and efficiency improvements to Theano, and benchmarks demonstrating Theano's performance relative to Torch7, a recently introduced machine learning library, and to RNNLM, a C++ library targeted at recurrent neural networks.

Subfields: Computational Theory and Mathematics, Hardware and Architecture, Statistical and Nonlinear Physics, Optimization & Training, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Theory and Mathematics

In Hardware and Architecture

In Statistical and Nonlinear Physics

In Optimization & Training

In Evaluation & Benchmarks

Citations in the corpus

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling (2014, background): “, Bengio et al. [1994] that it is difficult to train RNNs to capture long-term dependencies because the gradients tend to either vanish (most of the time) or explode (rarely, but with severe effects). This makes gradient-based optimization method struggle, not just because of the variations in gradi”
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation (2014, background): “In that case, for a given source phrase, the RNN Encoder–Decoder will need to generate a list of (good) target phrases.”
FitNets: Hints for Thin Deep Nets (2014, methodology): “ers. The hint-based training suggests that more efforts should be devoted to explore new training strategies to leverage the power of deep networks. ACKNOWLEDGMENTS We thank the developers of Theano (Bastien et al., 2012) and Pylearn2 (Goodfellow et al., 2013a) and the computational resources provid”
Pylearn2: a machine learning research library (2013, methodology): “This is possible because Pylearn2 is built on top of Th eano [5, 3].”
MADE: Masked Autoencoder for Distribution Estimation (2015, methodology): “We thank Marc-Alexandre Côté for helping to implement NADE in Theano and the whole Theano (Bastien et al., 2012; Bergstra et al., 2010) team of contributors.”
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020, background): “Software Software systems supporting neural networks evolved together with the advancement of the underlying hardware (Dean et al., 2012; Bastien et al., 2012; Abadi et al., 2016; Paszke et al., 2017; Palkar & Zaharia, 2019).”
Training Deep Nets with Sublinear Memory Cost (2016, methodology): “Here we use coarse grained forward and backward operations to make the graph simpler.”
NICE: Non-linear Independent Components Estimation (2014, methodology): “They are grateful towards the developers of Theano (Bergstra et al., 2011) (Bastien et al., 2012) and Pylearn2 (Goodfellow et al., 2013), and for the computational resources provided by Compute Canada and Calcul Québec, and for the research funding provided by NSERC, CIFAR, and Canada Research…”
Compatible Value Gradients for Reinforcement Learning of Continuous Deep Policies (2015, positive): “Both algorithms were implemented in Theano (Bergstra et al., 2010; Bastien et al., 2012).”
Convolutional Networks on Graphs for Learning Molecular Fingerprints (2015, methodology): “Software Automatic differentiation (AD) software packages such as Theano [1] significantly speed up development time by providing gradients automatically, but can only handle limited control structures and indexing.”
On the difficulty of training recurrent neural networks (2012, methodology): “We use Theano to compute these gradients (Bergstra et al. , 2010; Bastien et al. , 2012).”
Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations (2016, methodology): “In our Theano implementation we used hidden layers of size 4096 whereas in our Torch implementation we used much smaller size 2048.”
Value Iteration Networks (2016, methodology): “y end-to-end training. Once a VIN design is chosen, implementing the VIN is straightforward, as it is simply a form of a CNN. The networks in our experiments all required only several lines of Theano [1] code. In the next section, we evaluate VIN policies on various domains, showing that by learning”