Meta-Learning Update Rules for Unsupervised Representation Learning

By Luke Metz, Niru Maheswaranathan, Brian Cheung, Jascha Narain Sohl-Dickstein. 2018

132 citations recorded by the source corpus.

A major goal of unsupervised learning is to discover data representations that are useful for subsequent tasks, without access to supervised labels during training. Typically, this involves minimizing a surrogate objective, such as the negative log likelihood of a generative model, with the hope that representations useful for subsequent tasks will arise as a side effect. In this work, we propose instead to directly target later desired tasks by meta-learning an unsupervised learning rule which leads to representations useful for those tasks. Specifically, we target semi-supervised classification performance, and we meta-learn an algorithm -- an unsupervised weight update rule -- that produces representations useful for this task. Additionally, we constrain our unsupervised update rule to a be a biologically-motivated, neuron-local function, which enables it to generalize to different neural network architectures, datasets, and data modalities. We show that the meta-learned update rule produces useful features and sometimes outperforms existing unsupervised learning techniques. We further show that the meta-learned unsupervised update rule generalizes to train networks with different widths, depths, and nonlinearities. It also generalizes to train on data with randomly permuted input dimensions and even generalizes from image datasets to a text task.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

AutoML-Zero: Evolving Machine Learning Algorithms From Scratch (2020, background): “tures, such as a second neural network [73, 60], an LSTM [67], a hierarchical RNN [88], or even a different LSTM for each weight [53].”
Meta Pseudo Labels (2020, background): “An exception is “Learning Unsupervised Updates” (Metz et al., 2019), where the synthetic gradient is meta-learned via an explicit outer loop.”
Meta-Learning in Neural Networks: A Survey (2020, methodology): “Successful applications have been demonstrated in areas spanning few-shot image recognition [19], [23], unsupervised learning [16], data efficient [24], [25] and self-directed [26] reinforcement learning (RL), hyperparameter optimization [20], and neural architecture search (NAS) [21], [27], [28].”
An Ensemble of Epoch-Wise Empirical Bayes for Few-Shot Learning (2019, methodology): “Related methods using gradient descent mostly work for single model learning in an inductive way [3, 10, 15, 44, 46–49].”
Meta-Learning with Warped Gradient Descent (2019, background): “WarpGrad is designed for integrating learning paradigms, a promising avenue for future research [28, 27].”
When MAML Can Adapt Fast and How to Assist When It Cannot (2019, background): “[30] Luke Metz, Niru Maheswaranathan, Brian Cheung, and Jascha Sohl-Dickstein.”
Adaptive Risk Minimization: A Meta-Learning Approach for Tackling Group Shift (2020, methodology): “Some meta-learning methods adapt using both labeled and unlabeled data, such as [53, 71, 46, 2, 39, 4], or consider the setting where task groupings are not known, such as [23, 30, 24, 59], though these works do not focus on the same setting of distribution shift.”
Revisiting Meta-Learning as Supervised Learning (2020, methodology): “There are excellent overviews and surveys of metalearning (Thrun and Pratt, 1998; Vilalta and Drissi, 2002; Lemke et al., 2015; Vanschoren, 2018; Finn, 2018; Metz et al., 2018).”
What Can Learned Intrinsic Rewards Capture? (2019, background): “…that directly meta-learn how to update the agent’s parameters via meta-parameters including discount factor and returns (Xu et al., 2018b), auxiliary tasks (Schlegel et al., 2018; Veeriah et al., 2019), unsupervised learning rules (Metz et al., 2019), and RL objectives (Bechtle et al., 2019).”
Zero-shot Text Classification With Generative Language Models (2019, methodology): “Future extensions should investigate the ability of gradient based metalearning to adapt to task descriptors, either through K-shot support-based learning or by taking gradient steps on the task descriptors themselves as in Metz et al. (2018).”
DeepEMD: Differentiable Earth Mover's Distance for Few-Shot Learning (2020, unknown): “, [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42], [43], [44], [45], [46], [47], [48], [49], [50], [51], [52], [53], target at effectively adapting model parameters to new tasks in the low-shot regime.”
Learning to (Learn at Test Time): RNNs with Expressive Hidden States (2024, unknown): “In prior work such as [2], [21] and [55], the inner loop learns from an entire dataset at a time instead of a sequence, so the outer loop needs a collection of datasets or tasks.”
DeepEMD: Few-Shot Image Classification With Differentiable Earth Mover’s Distance and Structured Classifiers (2020, unknown): “Optimization-based methods [2, 5, 9–11, 18, 26, 31–33, 36, 38, 40–42, 44, 46, 47, 54, 55, 58, 64, 79] target at effectively adapting model parameters to new tasks in the low-shot regime.”
Understanding and correcting pathologies in the training of learned optimizers (2018, methodology): “Additionally, we can simplify the parametric form of the optimizer, utilizing a small MLP without any complex tricks previously needed for stability (Metz et al., 2018; Wichrowska et al., 2017).”
Meta-Learning Requires Meta-Augmentation (2020, background): “Some meta-learners utilize an inner-loop objective that bears no resemblance to the objective computed on the query set [10, 24], Contextual meta-learners replace explicit inner-loop optimization with information extraction using a neural network, which is then learnt jointly in the outer-loop…”
Contextualizing Multiple Tasks via Learning to Decompose (2021, background): “Initialization of deep neural network [30, 15], instance embeddings [41, 29, 45, 38, 19], optimization policies [35, 19] and data augmentation [56, 22] could be learned across sampled tasks.”