Why Do Similarity Matching Objectives Lead to Hebbian/Anti-Hebbian Networks?

By Cengiz Pehlevan, Anirvan M. Sengupta, Dmitri "Mitya" Chklovskii. 2017

86 citations recorded by the source corpus.

Modeling self-organization of neural networks for unsupervised learning using Hebbian and anti-Hebbian plasticity has a long history in neuroscience. Yet derivations of single-layer networks with such local learning rules from principled optimization objectives became possible only recently, with the introduction of similarity matching objectives. What explains the success of similarity matching objectives in deriving neural networks with local learning rules? Here, using dimensionality reduction as an example, we introduce several variable substitutions that illuminate the success of similarity matching. We show that the full network objective may be optimized separately for each synapse using local learning rules in both the offline and online settings. We formalize the long-standing intuition of the rivalry between Hebbian and anti-Hebbian rules by formulating a min-max optimization problem. We introduce a novel dimensionality reduction objective using fractional matrix exponents. To illustrate the generality of our approach, we apply it to a novel formulation of dimensionality reduction combined with whitening. We confirm numerically that the networks with learning rules derived from principled objectives perform better than those with heuristic learning rules.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. German Ignacio Parisi
2. Stefan Wermter
3. Cornelius Weber
4. Jun Tani
5. Lars Hiller Eidnes
6. Arild Nøkland
7. Vincenzo Lomonaco
8. Christopher Kanan

In Cognitive Neuroscience

1. German Ignacio Parisi
2. Stefan Wermter
3. Cornelius Weber
4. Arild Nøkland
5. Jun Tani
6. Lars Hiller Eidnes

In Electrical and Electronic Engineering

1. Arild Nøkland
2. Stefan Wermter
3. Jun Tani
4. Cornelius Weber
5. Lars Hiller Eidnes
6. German Ignacio Parisi

Citations in the corpus

• Lifelong Learning of Spatiotemporal Representations With Dual-Memory Recurrent Self-Organization (2018, background): “However, large-scale annotated data is expensive to obtain and annotations may not be available in more natural experimental settings (Pehlevan et al., 2018).”
• Training Neural Networks with Local Error Signals (2019, methodology): “An online version of the sim-bio loss can be implemented using local Hebbian and anti-Hebbian learning rules, at least for unsupervised learning (Pehlevan et al., 2018; Giovannucci et al., 2018).”