Separable Layers Enable Structured Efficient Linear Substitutions

By Gavia Gray, Elliot J. Crowley, Amos J. Storkey. 2019

3 citations recorded by the source corpus.

In response to the development of recent efficient dense layers, this paper shows that something as simple as replacing linear components in pointwise convolutions with structured linear decompositions also produces substantial gains in the efficiency/accuracy tradeoff. Pointwise convolutions are fully connected layers and are thus prepared for replacement by structured transforms. Networks using such layers are able to learn the same tasks as those using standard convolutions, and provide Pareto-optimal benefits in efficiency/accuracy, both in terms of computation (mult-adds) and parameter count (and hence memory). Code is available at this https URL.

Subfields: Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

1. Mikhail Khodak
2. Lester W. Mackey
3. Nicoló Fusi
4. Neil A. Tenenholtz

In Efficiency & Compression

1. Mikhail Khodak
2. Lester W. Mackey
3. Nicoló Fusi
4. Neil A. Tenenholtz

Citations in the corpus

• Initialization and Regularization of Factorized Neural Layers (2021, methodology): “We also compare to CRS (Gray et al., 2019), which scales regular weightdecay by the compression rate.”