On the Expressive Power of Deep Neural Networks

By Maithra Raghu, Ben Poole, Jon M. Kleinberg, Surya Ganguli, Jascha Narain Sohl-Dickstein. 2016

897 citations recorded by the source corpus.

We propose a new approach to the problem of neural network expressivity, which seeks to characterize how structural properties of a neural network family affect the functions it is able to compute. Our approach is based on an interrelated set of measures of expressivity, unified by the novel notion of trajectory length, which measures how the output of a network changes as the input sweeps along a one-dimensional path. Our findings can be summarized as follows: (1) The complexity of the computed function grows exponentially with depth. (2) All weights are not equal: trained networks are more sensitive to their lower (initial) layer weights. (3) Regularizing on trajectory length (trajectory regularization) is a simpler alternative to batch normalization, with the same performance.

Subfields: Artificial Intelligence, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Yoshua Bengio
  2. Mohit Bansal
  3. Quoc V. Le
  4. Jianfeng Gao
  5. Percy Liang
  6. Luke S. Zettlemoyer
  7. Sergey Levine
  8. Kyunghyun Cho

In Scaling & Learning Theory

  1. Greg Yang
  2. Sam Schoenholz
  3. Yuanzhi Li
  4. Jeffrey Pennington
  5. Boris Hanin
  6. Jascha Narain Sohl-Dickstein
  7. Simon Shaolei Du
  8. Quanquan Gu

Citations in the corpus