Putting An End to End-to-End: Gradient-Isolated Learning of Representations

By Sindy Löwe, Peter O'Connor, Bastiaan S. Veeling. 2019

168 citations recorded by the source corpus.

We propose a novel deep learning method for local self-supervised representation learning that does not require labels nor end-to-end backpropagation but exploits the natural order in data instead. Inspired by the observation that biological neural networks appear to learn without backpropagating a global error signal, we split a deep neural network into a stack of gradient-isolated modules. Each module is trained to maximally preserve the information of its inputs using the InfoNCE bound from Oord et al. [2018]. Despite this greedy training, we demonstrate that each module improves upon the output of its predecessor, and that the representations created by the top module yield highly competitive results on downstream classification tasks in the audio and visual domain. The proposal enables optimizing modules asynchronously, allowing large-scale distributed training of very deep neural networks on unlabelled datasets.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

Why Should We Add Early Exits to Neural Networks? (2020, background): “Alternative supervised and unsupervised training strategies have also been proposed in a number of papers [ 20, 45 ] .”
Data Augmenting Contrastive Learning of Speech Representations in the Time Domain (2020, methodology): “ing, the loss is contrastive: an embedding should be close to positive future embeddings and distant from negative future embeddings. CPC was used as pretraining for ASR [1] and speaker identiﬁcation [13, 14]. Non-contrastive versions of predictive coding with ﬁxed embeddings can learn generic multi”
Contrastive Representation Learning: A Framework and Review (2020, methodology): “In this direction, Greedy InfoMax (GIM) [65] extends the approach of CPC [77] while Local Contrastive (LoCo) [114] improved the performance by extending SimCLR [16] with a modified overlapping architecture between local layers.”
Contrastive Learning for Unpaired Image-to-Image Translation (2020, background): “Associated signals can be an image with itself [49,67,17,24,80], an image with its downstream representation [28,47], neighboring patches within an image [33,25,57], or multiple views of the input image [73], and most successfully, an image with a set of transformed versions of itself [9,54].”
The Forward-Forward Algorithm: Some Preliminary Investigations (2022, methodology): “• For spatial data, can FF benefit from having lots of local goodness functions for different regions of the image (Löwe et al., 2019)? If this can be made to work, it should allow learning to be much faster.”
Hard Negative Mixing for Contrastive Learning (2020, background): “Building on ideas from [30] where objectives are optimized locally, in LoCo [59] the authors propose to locally train overlapping blocks, effectively closing the performance gap between local learning and end-to-end contrastive learning algorithms.”
Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding (2021, background): “Contrastive Predictive Coding (Oord et al., 2018; Löwe et al., 2019) learns representations by predicting the future in latent space, eliminating the need to reconstruct the full input.”
Towards Unsupervised Learning of Speech Features in the Wild (2021, methodology): “The encoder architecture is the same (5 convolutional layers with kernel sizes [10,8,4,4,4], strides [5,4,2,2,2] and hidden dimension 256), for the context network, we used 2-layer LSTM, and for the prediction network, we used a multi-head transformer [35], each of the 12 heads predicting one…”
Seed the Views: Hierarchical Semantic Alignment for Contrastive Representation Learning (2022, background): “GIM [39] and LoCo [40] propose to train the neural networks in an asynchronously and decoupled way.”