What shapes feature representations? Exploring datasets, architectures, and training

By Katherine L. Hermann, Andrew Kyle Lampinen. 2020

179 citations recorded by the source corpus.

In naturalistic learning problems, a model's input contains a wide range of features, some useful for the task at hand, and others not. Of the useful features, which ones does the model use? Of the task-irrelevant features, which ones does the model represent? Answers to these questions are important for understanding the basis of models' decisions, as well as for building models that learn versatile, adaptable representations useful beyond the original training task. We study these questions using synthetic datasets in which the task-relevance of input features can be controlled directly. We find that when two features redundantly predict the labels, the model preferentially represents one, and its preference reflects what was most linearly decodable from the untrained model. Over training, task-relevant features are enhanced, and task-irrelevant features are partially suppressed. Interestingly, in some cases, an easier, weakly predictive feature can suppress a more strongly predictive, but more difficult one. Additionally, models trained to recognize both easy and hard features learn representations most similar to models that use only the easy feature. Further, easy features lead to more consistent representations across model runs than do hard features. Finally, models have greater representational similarity to an untrained model than to models trained on a different task. Our results highlight the complex processes that determine which features a model represents.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning explanations that are hard to vary (2020, background): “This motivates our notion of invariant mechanisms, and inspired related notions which have been proposed for robust regression (Rojas-Carulla et al., 2018; Heinze-Deml et al., 2018; Arjovsky et al., 2019; Hermann and Lampinen, 2020; Ahuja et al., 2020; Krueger et al., 2020).”
Shape or Texture: Understanding Discriminative Features in CNNs (2021, result): “In contrast, recent works (Brendel & Bethge, 2019; Hermann & Lampinen, 2020) have argued that CNNs trained on ImageNet (IN) (Deng et al., 2009) classify images mainly according to their texture , rather than object shape .”
Tell me why! - Explanations support learning of relational and causal structure (2021, background): “Hermann and Lampinen (2020) show similar feature-difﬁculty rankings for CNNs, and that CNNs lazily prefer easier features.”
The Pitfalls of Simplicity Bias in Neural Networks (2020, background): “[31] use color, shape and texture features in stylized settings to show that standard training procedures can (a) increase reliance on taskrelevant features that are partially decodable using untrained networks and (b) suppress reliance on non-discriminative or correlated features.”
Tell me why! Explanations support learning relational and causal structure (2021, background): “These results concord with the features that Hermann & Lampinen (2020) found were easiest for CNNs and ResNets to learn, suggesting that explanations may help overcome the preference of agents (or other networks) to be “lazy” and prefer “shortcut features” (Geirhos et al., 2020).”
Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth (2020, background): “…from properties of neural network training (Gotmare et al., 2018; Neyshabur et al., 2020), objectives (Resnick et al., 2019; Thompson et al., 2019; Hermann & Lampinen, 2020), and dynamics (Maheswaranathan et al., 2019) to revealing hidden linguistic structure in large language models (Bau et al.,…”
Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations (2022, background): “Hermann and Lampinen [38] and Jacobsen et al.”
Gradient Starvation: A Learning Proclivity in Neural Networks (2020, background): “Recent work aims at identifying the reasons behind this phenomenon [97, 70, 22, 73, 51, 76, 100, 92, 83, 105, 42, 79, 4], while complementary work quantifies resulting shortcomings, including poor generalization to out-of-distribution (OOD) test data, reliance upon spurious correlations, and lack of”
Dataset Distillation with Infinitely Wide Convolutional Networks (2021, background): “Such resulting datasets will not arise from the distribution of natural images but will nevertheless capture features useful to a neural network, a capability which remains mysterious and is far from being well-understood [Ilyas et al., 2019, Huh et al., 2016, Hermann and Lampinen, 2020].”
Transforming task representations to perform novel tasks (2020, background): “Performing a meta-mapping from a representation: Given a meta-mapping representation z meta and an input task representation z task , the transformed task representation would be computed as: Varies.”