Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth

By Thao Nguyen, Maithra Raghu, Simon Kornblith. 2020

340 citations recorded by the source corpus.

A key factor in the success of deep neural networks is the ability to scale models to improve performance by varying the architecture depth and width. This simple property of neural network design has resulted in highly effective architectures for a variety of tasks. Nevertheless, there is limited understanding of effects of depth and width on the learned representations. In this paper, we study this fundamental question. We begin by investigating how varying depth and width affects model hidden representations, finding a characteristic block structure in the hidden representations of larger capacity (wider or deeper) models. We demonstrate that this block structure arises when model capacity is large relative to the size of the training set, and is indicative of the underlying layers preserving and propagating the dominant principal component of their representations. This discovery has important ramifications for features learned by different models, namely, representations outside the block structure are often similar across architectures with varying widths and depths, but the block structure is unique to each model. We analyze the output predictions of different model architectures, finding that even when the overall accuracy is similar, wide and deep models exhibit distinctive error patterns and variations across classes.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Universal Representation Learning from Multiple Domains for Few-shot Classification (2021, methodology): “As the original CKA similarity requires the computation of the kernel matrices over the whole datasets, which is not scalable to large datasets, we follow [34] and com-”
How Do Vision Transformers Work? (2022, methodology): “Feature map similarities of CNNs shows a block structure (Nguyen et al., 2021).”
Why Do Better Loss Functions Lead to Less Transferable Features? (2020, methodology): “Centered kernel alignment (CKA) reveals large differences in representations of the last few layers of the network, whereas earlier layers are similar regardless of which training objective is used.”
Do Self-Supervised and Supervised Methods Learn Similar Visual Representations? (2021, methodology): “Prior work has demonstrated the utility of Centered Kernel Alignment (CKA) as a similarity index which elegantly addresses these challenges [16], enabling the analysis of a variety of neural architectures [16, 17, 18].”
Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference (2024, neutral): “At the same parameter count, models with more narrow layers ( Deep & Narrow ) have different learning patterns than models with fewer wide layers ( Shallow & Wide ) (Nguyen et al., 2021).”
Do Vision Transformers See Like Convolutional Neural Networks? (2021, methodology): “To work at scale with our models and tasks, we approximate the unbiased estimator of HSIC [39] using minibatches, as suggested in [29].”
Git Re-Basin: Merging Models modulo Permutation Symmetries (2022, background): “Prior works noted the feature/layer association (Nguyen et al., 2021) and re-scaling invariances (Ainsworth et al., 2018).”