The Geometry of Deep Networks: Power Diagram Subdivision
By Randall Balestriero, Romain Cosentino, Behnaam Aazhang, Richard Baraniuk. 2019
77 citations recorded by the source corpus.
We study the geometry of deep (neural) networks (DNs) with piecewise affine and convex nonlinearities. The layers of such DNs have been shown to be {\em max-affine spline operators} (MASOs) that partition their input space and apply a region-dependent affine mapping to their input to produce their output. We demonstrate that each MASO layer's input space partitioning corresponds to a {\em power diagram} (an extension of the classical Voronoi tiling) with a number of regions that grows exponentially with respect to the number of units (neurons). We further show that a composition of MASO layers (e.g., the entire DN) produces a progressively subdivided power diagram and provide its analytical form. The subdivision process constrains the affine maps on the (exponentially many) power diagram regions to greatly reduce their complexity. For classification problems, we obtain a formula for a MASO DN's decision boundary in the input space plus a measure of its curvature that depends on the DN's nonlinearities, weights, and architecture. Numerous numerical experiments support and extend our theoretical results.
Subfields: Computational Mechanics, Computational Theory and Mathematics, Scaling & Learning Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computational Mechanics
- Pascal Vincent
- Florian Bordes
- Yoshua Bengio
- Yang Song
- Stefano Ermon
- Aaron C. Courville
- Ferenc Huszár
- Diederik P. Kingma
In Computational Theory and Mathematics
In Scaling & Learning Theory
Citations in the corpus
- High Fidelity Visualization of What Your Self-Supervised Representation Knows About (2021, background): “In fact, the regions ω ∈ Ω are often extremely localized in the input space, especially as the architecture involves many layers (Mont ´ ufar et al., 2014; Balestriero et al., 2019).”