Depth Separation in ReLU Networks for Approximating Smooth Non-Linear Functions

By Itay Safran, Ohad Shamir. 2016

32 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Deep Neural Networks for Estimation and Inference: Application to Causal Effects and Other Semiparametric Estimands (2018, background): “Comprehensive theoretical treatments are given by White (1992) and Anthony and Bartlett (1999).”
Mixing Complexity and its Applications to Neural Networks (2017, background): “This has led many researchers to attempt to understand the reasons for the astonishing success despite the proven hardness (Shamir, 2016; Safran & Shamir, 2016; Eldan & Shamir, 2016; Daniely et al., 2016; Raghu et al., 2016; Arora et al., 2014; Livni et al., 2014).”
Nearly-tight VC-dimension bounds for piecewise linear neural networks (2017, background): “[11] show that a sufficiently non-linear C(2) function on [0, 1] can be approximated with error in L2 by a ReLU network with O(polylog(1/ )) layers and weights, but any such approximation with O(1) layers requires Ω(1/ ) weights.”
Learning Neural Networks with Two Nonlinear Layers in Polynomial Time (2017, background): “Following Safran and Shamir (2016), we define a neural network with one (nonlinear) layer with k units as N1 : x → ∑k i=1 biσ(ai · x) for x ∈ Rn, ai ∈ Sn−1 for i ∈ {1, · · · , k}, b ∈ Sk−1.”
Depth Separation for Neural Networks (2017, background): “In particular, separation for deep networks [11, 10], expressive power of depth two networks [4, 8, 7, 2], and more [5, 3].”
Why and when can deep-but not shallow-networks avoid the curse of dimensionality: A review (2016, background): “In fact, examples of speciﬁc functions that cannot be represented efﬁciently by shallow networks have been given very recently by Telgarsky [25] and by Shamir [26] .”
Optimal approximation of continuous functions by very deep ReLU networks (2018, background): “Depth-separation results for some explicit families of functions have been obtained in Safran and Shamir (2016); Telgarsky (2016).”
On the Expressive Power of Overlapping Architectures of Deep Learning (2017, methodology): “…Hajnal et al., 1993), and the more recent studies covering the types of networks used in practice (Pascanu et al., 2013; Montúfar et al., 2014; Eldan and Shamir, 2016; Cohen et al., 2016a; Cohen and Shashua, 2016; Telgarsky, 2016; Safran and Shamir, 2016; Raghu et al., 2016; Poole et al., 2016).”