Wider Networks Learn Better Features

By Dar Gilboa, Guy Gur-Ari. 2019

7 citations recorded by the source corpus.

Transferability of learned features between tasks can massively reduce the cost of training a neural network on a novel task. We investigate the effect of network width on learned features using activation atlases --- a visualization technique that captures features the entire hidden state responds to, as opposed to individual neurons alone. We find that, while individual neurons do not learn interpretable features in wide networks, groups of neurons do. In addition, the hidden state of a wide network contains more information about the inputs than that of a narrow network trained to the same test accuracy. Inspired by this observation, we show that when fine-tuning the last layer of a network on a new task, performance improves significantly as the width of the network is increased, even though test accuracy on the original task is independent of width.

Subfields: Artificial Intelligence, Model Architectures, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Scaling & Learning Theory

Citations in the corpus

Feature Learning in Infinite-Width Neural Networks (2020, background): “[14] empirically observed that wider networks achieve better downstream performance with linear transfer learning, even though on the original pretraining task there can be little difference.”
Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics (2020, background): “Inspired by recent work [43] showing that wider networks can learn better features than their narrower counterparts, we sought to determine whether wider networks also exhibit less catastrophic forgetting than narrower ones.”