On the number of inference regions of deep feed forward networks with piece-wise linear activations

By Razvan Pascanu, Guido Montúfar, Yoshua Bengio. 2013

84 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

On the Number of Linear Regions of Deep Neural Networks (2014, methodology): “This implies that the maximal number of linear regions of functions computed by a shallow rectiﬁer network with n 0 inputs and n 1 hidden units is n 0 j =0 n 1 j (see Pascanu et al. 2013; Proposition 5).”
Convergence Analysis of Two-layer Neural Networks with ReLU Activation (2017, background): “ReLU is the state-ofthe-art activation function [30, 13], and has great expressive power as well [29, 32, 31, 4, 26].”
Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learning (2014, background): “On the other hand, there are both theoretical and empirical indications suggesting that deep distributed representations (Bengio, 2009; Pascanu et al. , 2014) can beneﬁt from advantageous statistical properties, when the data has been generated by multiple factors organized hierarchically, with the…”
Inductive biases for deep learning of higher-level cognition (2020, background): “Interestingly, different inductive biases may be equivalent to more or less data (even possibly exponentially more data): we suspect that inductive biases based on a form of compositionality (like distributed representations [125], depth [126] and attention [6,127]) can potentially also provide a la”
Inductive Bias of Deep Convolutional Networks through Pooling Geometry (2016, background): “In recent years, a large body of research was devoted to proving existence of depth efﬁciency under different types of architectures (see for example [9, 23, 21, 28, 10, 24, 20]).”
On simplicity and complexity in the brave new world of large-scale neuroscience. (2015, background): “The theory of deep learning is still in its infancy, but some examples of general statements one can make about deep circuits without actually simulating them include how their functional complexity scales with depth [70, 71], how their synaptic weights, over time, acquire statistical structures in ”
Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons (2018, methodology): “(Montufar et al. 2014; Pascanu, Mont´ufar, and Bengio 2014; Pan and Srikumar 2016).”
Bounding and Counting Linear Regions of Deep Neural Networks (2017, methodology): “Given a PWL function F : Rn0 → Rm represented by a DNN, a linear region is the set of inputs that correspond to a same activation pattern in the DNN. 1There is a subtly different definition of linear region in the literature (Pascanu et al., 2014; Montúfar et al., 2014), as follows.”
Bounds on the Approximation Power of Feedforward Neural Networks (2018, background): “…ones, given a fixed number of hidden units (Bianchini & Scarselli, 2014; Delalleau & Bengio, 2011; Liang & Srikant, 2017; Mhaskar et al., 2016; Pascanu et al., 2014; Telgarsky, 2015; 2016; Yarotsky, 2017).1 A popular activation function is the rectified linear unit (ReLU), partly because of…”
Neural Taylor Approximations: Convergence and Exploration in Rectifier Networks (2016, methodology): “The success of these methods on rectiﬁer networks, where the number of kinks grows exponentially with depth (Pascanu et al., 2014; Telgarsky, 2016), requires explanation since gradients at nearby points can be very different (Balduzzi et al., 2017).”
On the Expressive Power of Deep Learning: A Tensor Analysis (2015, unknown): “Pascanu et al. (2013) and Montufar et al. (2014) study the number of linear regions that may be expressed by such networks as a function of their depth and width, thereby showing existence of functions separating deep from shallow (depth 2) networks.”
How Can Deep Rectifier Networks Achieve Linear Separability and Preserve Distances? (2015, background): “Similarly, with the same number of hidden units, deep networks are able to separate their input space into much more regions of linearity than their shallow counterparts (Pascanu et al., 2014; Montúfar et al., 2014).”
Contractive Rectifier Networks for Nonlinear Maximum Margin Classification (2015, background): “The advantages of deep rectifier networks are not only shown in their excellent empirical performance in object recognition [18, 20, 13], face verification [38, 37], speech recognition ([32, 14, 7] and handwritten digit recognition [5], but also justified by a number of theoretical works on their su”