Theano: A CPU and GPU Math Compiler in Python

By James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph P. Turian, David Warde-Farley, Yoshua Bengio. 2010

862 citations recorded by the source corpus.

Theano is a compiler for mathematical expressions in Python that combines the convenience of NumPy's syntax with the speed of optimized native machine language. The user composes mathematical expressions in a high-level description that mimics NumPy's syntax and semantics, while being statically typed and functional (as opposed to imperative). These expressions allow Theano to provide symbolic differentiation. Before performing computation, Theano optimizes the choice of expressions, translates them into C++ (or CUDA for GPU), compiles them into dynamically loaded Python modules, all automatically. Common machine learn- ing algorithms implemented with Theano are from 1:6 to 7:5 faster than competitive alternatives (including those implemented with C/C++, NumPy/SciPy and MATLAB) when compiled for the CPU and between 6:5 and 44 faster when compiled for the GPU. This paper illustrates how to use Theano, outlines the scope of the compiler, provides benchmarks on both CPU and GPU processors, and explains its overall design.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Hardware and Architecture, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Hardware and Architecture

In ML Systems & Hardware

Citations in the corpus

AllenNLP: A Deep Semantic Natural Language Processing Platform (2018, background): “Most existing deep-learning toolkits are designed for general machine learning (Bergstra et al., 2010; Yu et al., 2014; Chen et al., 2015; Abadi et al., 2016; Neubig et al., 2017), and can require signiﬁ-cant effort to develop research infrastructure for particular model classes.”
Learning how to explain neural networks: PatternNet and PatternAttribution (2017, methodology): “We used Theano (Bergstra et al., 2010) and Lasagne (Dieleman et al., 2015) for our implementation.”
Accurate De Novo Prediction of Protein Contact Map by Ultra-Deep Learning Model (2016, methodology): “Existing deep learning development toolkits such as Theano (http://deeplearning.net/software/theano/) and Tensorflow (https://www.tensorflow.org/) have provided an API (application programming interface) for convolutional operation so that we do not need to implement it by ourselves.”
Variational Dropout Sparsifies Deep Neural Networks (2017, methodology): “We will provide a reference implementation using Theano (Bergstra et al., 2010) and Lasagne in the camera-ready version.”
Practical Recommendations for Gradient-Based Training of Deep Architectures (2012, methodology): “This is the strategy adopted in the Theano library11 with its Op objects (Bergstra et al., 2010), as well as in libraries such as Torch12 (Collobert et al., 2011b) and Lush13.”
Visual Causal Feature Learning (2014, background): “0014-10-1-0933. FE would like to thank Cosma Shalizi for the introduction to computational mechanics and many relevant results this paper builds on. KC would like to thank the creators of the Theano (Bergstra et al., 2010; Bastien et al., 2012) and Pylearn2 (Goodfellow et al., 2013) Python packages ”
Transformation Properties of Learned Visual Representations (2014, methodology): “The gradients of which are easily computed using backpropagation (in our own implementation, we compute gradients automatically using Theano (Bergstra et al., 2010)).”
Deep Learning of Representations: Looking Forward (2013, background): “They have also taken advantage of GPU technology to speed-up computation by one or two orders of magnitude (Raina et al., 2009; Bergstra et al., 2010, 2011; Krizhevsky et al., 2012).”
Ray: A Distributed Framework for Emerging AI Applications (2017, methodology): “rdparty libraries. These libraries include simulators such as OpenAI gym [13], DeepMind Lab [10], and the Mujoco physics simulator [45] as well as deep learning frameworks like TensorFlow [5], Theano [12], PyTorch [4], and Caffe [26]. As we will see, this requires augmenting the task-parallel model ”
Deep UQ: Learning deep neural network surrogate models for high dimensional uncertainty quantification (2018, methodology): “Development of easy-to-use software libraries, such as Tensorﬂow [53], PyTorch [54] and Theano [55].”